The Sage Handbook of Mixed Methods Research Design [1 ed.] 1529723965, 9781529723960

Citation preview

本书版权归SAGE所有

The Sage Handbook of

Mixed Methods Research Design

Editorial Section Leads Each of the six sections of the Sage Handbook of Mixed Methods Research Design has benefited from being led by talented section leads. The 13 section leads are established authors and esteemed contributors to the global mixed methods research community. The section leads have worked closely and tirelessly with the editor and supporting authors to bring to life the ideas first advanced in their proposals, further developed in their initial chapter drafts, and then refined with peer review and editorial feedback. We thank them heartily for their wisdom, commitment, and enthusiasm! Section 1: Inspiring Diversity and Innovation in Mixed Methods Design Sergi Fàbregues, Universitat Oberta de Catalunya, Spain José F. Molina-Azorin, University of Alicante, Spain Section 2: The Craft of Mixed Methods Design Judith Schoonenboom, University of Vienna, Austria Sophia L. Johnson, USA Section 3: Expanding Mixed Methods Design Approaches Peggy Shannon-Baker, Georgia Southern University, USA Jessica T. DeCuir-Gunby, University of Southern California, USA Section 4: Designing Innovative Integrations with Technology Timothy C. Guetterman, University of Michigan, USA Section 5: Navigating Research Cultures in Mixed Methods Design Elizabeth G. Creamer, Virginia Tech University, USA Elsa Lucia Escalante-Barrios, Universidad del Norte, Colombia Section 6: Exploring Design Possibilities and Challenges for Mixed Methods Research for the Future Peter Rawlins, Massey University, New Zealand Maggie Hartnett, Massey University, New Zealand

International Advisory Board The Sage Handbook of Mixed Methods Research Design has benefited from the contributions and insights of an International Advisory Board. The 12 Advisory Board members represent a range of established and emerging scholars from varied disciplines and geographical locations. The members have played vital roles in community-sourcing suggestions for chapter topics and authors, in providing constructive feedback to authors, and in some cases, lending support to chapter contributors as coaches. We are deeply appreciative of their investment of time and expertise to bring this handbook to life. Mandy M. Archibald, University of Manitoba, Canada Lisbeth M. Brevik, University of Oslo, Norway Roslyn Cameron, Torrens University Australia, Australia Loraine D. Cook, University of West Indies, Mona Campus, Jamaica John W. Creswell, University of Michigan, USA Jenny Douglas, Open University, UK Michael D. Fetters, University of Michigan, USA Taichi Hatta, Shizuoka Graduate University of Public Health, Japan Tera R. Jordan, Iowa State University, USA Donna M. Mertens, Gallaudet University in Washington, DC, USA Katrin Niglas, Tallinn University, Estonia Vanessa Scherman, University of South Africa, South Africa

This page intentionally left blank

The Sage Handbook of

Mixed Methods Research Design

Edited by

Cheryl N. Poth

1 Oliver’s Yard 55 City Road London EC1Y 1SP 2455 Teller Road Thousand Oaks, California 91320 Unit No 323-333, Third Floor, F-Block International Trade Tower Nehru Place New Delhi 110 019 8 Marina View Suite 43-053 Asia Square Tower 1 Singapore 018960

Editor: Umeeka Raichura Assistant Editor: Colette Wilson Production Editor: Neelu Sahu Copyeditor: Diana Chambers Proofreader: Brian McDowell Indexer: KnowledgeWorks Global Ltd Marketing Manager: Ben Sherwood Cover Design: Ginkhan Siam Typeset by KnowledgeWorks Global Ltd Printed in the UK

Editorial Arrangement & Introduction © Cheryl N. Poth, 2023 Section 1 Introduction © Sergi Fàbregues and José F. Molina-Azorin, 2023 Chapter 2 © John W. Creswell and Vicki L. Plano Clark, 2023 Chapter 3 © Joseph A Maxwell, 2023 Chapter 4 © Donna M. Mertens, 2023 Chapter 5 © Katrin Niglas, 2023 Chapter 6 © Dawn Freshwater and Jane Cahill, 2023 Section 1 Conclusion © José F. Molina-Azorin & Sergi Fàbregues, 2023 Section 2 Introduction © Sophia L. Johnson and Judith Schoonenboom, 2023 Chapter 7 © Manuela De Allegri and Julia Lohmann, 2023 Chapter 8 © Judith Schoonenboom, 2023 Chapter 9 © Julie A. Corrigan and Anthony J. Onwuegbuzie, 2023 Chapter 10 © Susanne Vogl, 2023 Chapter 11 © Roslyn Cameron and Heinz Herrmann, 2023 Chapter 12 © Elizabeth G. Creamer, Cassandra McCall and Cherie D. Edwards, 2023 Section 2 Conclusion © Judith Schoonenboom and Sophia L. Johnson, 2023 Section 3 Introduction © Peggy Shannon-Baker and Jessica T. DeCuir-Gunby, 2023 Chapter 13 © Jenny Douglas, 2023 Chapter 14 © Mehdi Taghipoorreyneh, 2023 Chapter 15 © Peter Rawlins, Philippa Butler, Spencer Lilley and Maggie Hartnett, 2023 Chapter 16 © Tera R. Jordan and Maya Bartel, 2023 Chapter 17 © Jenevieve Mannell and Audrey Prost, 2023 Chapter 18 © Joanne Mayoh, Talia Thompson and Shanlee Davis, 2023 Chapter 19 © Loraine D. Cook and Vimala Judy Kamalodeen, 2023 Chapter 20 © Vanessa Scherman and Lisa Zimmerman, 2023

Chapter 21 © Michelle C. Howell, Wayne A. Babchuk and Timothy C. Guetterman, 2023 Section 3 Conclusion © Jessica T. DeCuir-Gunby and Peggy ShannonBaker, 2023 Section 4 Introduction © Timothy C. Guetterman, 2023 Chapter 22 © Udo Kuckartz and Stefan Rädiker, 2023 Chapter 23 © Mitsuyuki Inaba and Hisako Kakai, 2023 Chapter 24 © Lisbeth M. Brevik, 2023 Chapter 25 © Daphne C. Watkins and Natasha C. Johnson, 2023 Chapter 26 © Carolina Bustamante, 2023 Section 4 Conclusion © Timothy Guetterman, 2023 Section 5 Introduction © Elizabeth Creamer, 2023 Chapter 27 © Jori N. Hall and Ayesha S. Boyce, 2023 Chapter 28 © Taichi Hatta, 2023 Chapter 29 © P. Paul Chandanabhumma, Annika Agni and Melissa DeJonckheere, 2023 Chapter 30 © Hongling Chu, Xuejun Yin and Hueiming Liu, 2023 Chapter 31 © M. Teresa Anguera, Eulàlia Arias-Pujol, Francisco Molinero and Luca Del Giacco, 2023 Section 5 Conclusion © Elsa Lucia Escalante Barrios, 2023 Section 6 Introduction © Peter Rawlins and Maggie Hartnett, 2023 Chapter 32 © Peggy ShannonBaker, 2023 Chapter 33 © José F. Molina-Azorin and Michael D. Fetters, 2023 Chapter 34 © Nataliya V. Ivankova, Jami L. Anderson, Ivan I. Herbey, Linda A. Roussel and Daniel Kim, 2023 Chapter 35 © Mandy M. Archibald, 2023 Chapter 36 © John W. Creswell, Cheryl N. Poth and Peter Rawlins, 2023 Section 6 Conclusion © Peter Rawlins and Maggie Hartnett, 2023 Chapter 37 Handbook Conclusion © Cheryl N. Poth, 2023

Apart from any fair dealing for the purposes of research, private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act, 1988, this publication may not be reproduced, stored or transmitted in any form, or by any means, without the prior permission in writing of the publisher, or in the case of reprographic reproduction, in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publisher. At Sage we take sustainability seriously. Most of our products are printed in the UK using responsibly sourced papers and boards. When we print overseas we ensure sustainable papers are used as measured by the Paper Chain Project grading system. We undertake an annual audit to monitor our sustainability.

Library of Congress Control Number: 2023942652 British Library Cataloguing in Publication data A catalogue record for this book is available from the British Library ISBN 978-1-5297-2396-0

For ALL the researchers around the world who continue to enrich our mixed methods research community and field with their cultural diversity and design innovations. Dare to follow your own paths. Dare to be yourself. Dare to be different. — Lailah Gifty Akita Author and Founder of Smart Youth Volunteers Foundation In memory of our esteemed colleague and chapter contributor Dr. Vimala J. Kamalodeen who touched many lives, and brought great energy and vision to the global mixed methods research community.

This page intentionally left blank

Contents List of Figures List of Tables List of Box Notes on the Editor, Section Leads, and Chapter Contributors Preface Acknowledgements 1

Dilemmas and Opportunities for Mixed Methods Research Design: Handbook Introduction Cheryl N. Poth

xiii xvi xviii xix xxxii xxxiii

1

SECTION 1  INSPIRING DIVERSITY AND INNOVATION IN MIXED METHODS DESIGN

Evolving Tensions and Conversations in Mixed Methods Research Design Approaches: Section 1 Introduction Sergi Fàbregues and José F. Molina-Azorin

2

Revisiting Mixed Methods Research Designs Twenty Years Later John W. Creswell and Vicki L. Plano Clark

21

3

Mixed Methods Design in Historical Perspective: Implications for Researchers Joseph A. Maxwell

37

4

Mixed Methods Designs to Further Social, Economic and Environmental Justice Donna M. Mertens

48

5

Developments in Mixed Methods Designs: What Have Been the Dominant Pathways and Where Might They Take Us in the Future? Katrin Niglas

59

6

The Role of Methodological Paradigms for Dialogic Knowledge Production: Using a Conceptual Map of Discourse Development to Inform Mixed Methods Research Design Dawn Freshwater and Jane Cahill

79



Future Tensions and Design Conversations in the Mixed Methods Field: Section 1 Conclusions José F. Molina-Azorin and Sergi Fàbregues

91

17

SECTION 2  THE CRAFT OF MIXED METHODS RESEARCH DESIGN

The Craft of Mixed Methods Research Design: Section 2 Introduction Sophia L. Johnson and Judith Schoonenboom

7

Embracing Emergence in Mixed Methods Designs: Theoretical Foundations and Empirical Applications Manuela De Allegri and Julia Lohmann

97

101

x

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

8

The Methods-Inference Map: Visualizing the Interactions Between Methods and Inferences in Mixed Methods Research Judith Schoonenboom

9

Towards Sampling Designs that are Transparent, Rigorous, Ethical and Equitable (TREE): Using a Tree Metaphor as a Sampling Meta-Framework in Mixed Methods Research Julie A. Corrigan and Anthony J. Onwuegbuzie

10

Data Integration as a Form of Integrated Mixed Analysis in Mixed Methods Research Designs Susanne Vogl

11

Ethical Issues and Practices for Mixed Methods Research in an Era of Big Data Roslyn Cameron and Heinz Herrmann

12

Building the Logic for an Integrated Methodology: Mixed Method Grounded Theory as an Example of Constructing a Methodology to Guide Design and Integration Elizabeth G. Creamer, Cassandra McCall and Cherie D. Edwards



The Craft of Mixed Methods Research Design: Section 2 Conclusions Judith Schoonenboom and Sophia L. Johnson

114

130

143 154

166 180

SECTION 3  EXPANDING MIXED METHODS DESIGN APPROACHES

Expanding Beyond Typology-Based Mixed Methods Designs: Section 3 Introduction Peggy Shannon-Baker and Jessica T. DeCuir-Gunby

187

13

Exploring Interlocking Relationships of Race, Gender, and Class with an Intersectionality-Informed Mixed Methods Research Design Framework Jenny Douglas

191

14

Indigenous Cultural Values Instrument Development: Using Mixed Methods Research Mehdi Taghipoorreyneh

203

15

What Can Mixed Methods Partnerships Learn from Kaupapa Māori Research Principles? Peter Rawlins, Philippa Butler, Spencer Lilley and Maggie Hartnett

218

16

Prioritizing Cultural Responsiveness in Mixed Methods Research and Team Science with Underrepresented Communities Tera R. Jordan and Maya Bartel

233

17

Using Participatory Methods in Randomised Controlled Trials of Complex Interventions Jenevieve Mannell and Audrey Prost

245

18

Illustrating the Mixed Methods Phenomenological Approach (MMPR) Joanne Mayoh, Talia Thompson and Shanlee Davis

19

Intersection of Mixed Methods and Case Study Research (MM+CSR): Two Design Options in Educational Research Loraine D. Cook and Vimala Judy Kamalodeen

256

267

Contents

xi

20

Harnessing Mixed Methods for Research Instrument Development and Legitimation Vanessa Scherman and Lisa Zimmerman

278

21

Mixed Methods-Grounded Theory: Best Practices for Design and Implementation Michelle C. Howell, Wayne A. Babchuk and Timothy C. Guetterman

291



Moving Beyond Tradition: The Need for Expanded and Culturally Relevant Mixed Methods Design Typologies: Section 3 Conclusions Jessica T. DeCuir-Gunby and Peggy Shannon-Baker

305

SECTION 4  DESIGNING INNOVATIVE INTEGRATIONS WITH TECHNOLOGY

Expanding Innovative Integrations with Technology: Section 4 Introduction Timothy C. Guetterman

311

22

Using Software for Innovative Integration in Mixed Methods Research: Joint Displays, Insights and Inferences with MAXQDA Udo Kuckartz and Stefan Rädiker

315

23

Grounded Text Mining Approach: An Integration Strategy of Grounded Theory and Textual Data Mining Mitsuyuki Inaba and Hisako Kakai

328

24

A “Mixed Methods Way of Thinking” in Game-based Research Integrations Lisbeth M. Brevik

25

Integrating Secondary Data from Ethnically and Racially Minoritized Groups in Mixed Methods Research Daphne C. Watkins and Natasha C. Johnson

361

26

Beyond the Joint Display in Mixed Methods Convergent Designs: A Case-Oriented Merged Analysis Carolina Bustamante

372



The Untapped Potential of Technology for Integration: Section 4 Conclusions Timothy C. Guetterman

346

387

SECTION 5  NAVIGATING RESEARCH CULTURES IN MIXED METHODS DESIGN

From Margin to Center: The Design Implications of a Cultural Component in Mixed Methods Research: Section 5 Introduction Elizabeth G. Creamer

27

Culturally Responsive Mixed Methods Evaluation Design Jori N. Hall and Ayesha S. Boyce

28

Integrating a Four-Step Japanese Cultural Narrative Framework, Ki-Shou-Ten-Ketsu, into a Mixed Methods Study Taichi Hatta

29

Leveraging Mixed Methods Community-based Participatory Research (MMCBPR) in Diverse Social and Cultural Contexts to Advance Health Equity

393 397

411

422

xii

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

30

Cultural Diversity in Intervention Designs: A Chinese Illustrative Example Hongling Chu, Xuejun Yin and Hueiming Liu

434

31

Examining the Influences of Spanish Research Culture in Systematic Observation with Mixed Methods M. Teresa Anguera, Eulàlia Arias-Pujol, Francisco Molinero and Luca Del Giacco

446



Future Direction for Navigating Research Cultures in Designs: Section 5 Conclusions Elsa Lucia Escalante-Barrios

463

SECTION 6  EXPLORING DESIGN POSSIBILITIES AND CHALLENGES FOR MIXED METHODS RESEARCH

Exploring Possibilities and Challenges for Mixed Methods Research for the Future: Section 6 Introduction Peter Rawlins and Maggie Hartnett

32

Visualizing the Process: Using Visuals to Teach and Learn Mixed Methods Research Peggy Shannon-Baker

33

Towards the Future Legitimacy of Mixed Methods Designs: Responsible Mixed Methods Research for Tackling Grand Challenges for the Betterment of Society José F. Molina-Azorin and Michael D. Fetters

34

Realizing Methodological Potentials and Advantages of Mixed Methods Research Design for Knowledge Translation Nataliya V. Ivankova, Jami L. Anderson, Ivan I. Herbey, Linda A. Roussel and Daniel Kim

35

Opportunities and Challenges for a Transdisciplinary Mixed Methods Research Future Mandy M. Archibald

512

36

Mapping Design Trends and Evolving Directions Using the Sage Handbook of Mixed Methods Research Design John W. Creswell, Cheryl N. Poth and Peter Rawlins

527



Where to Next in Exploring Possibilities and Challenges for Mixed Methods Research for the Future? Section 6 Conclusions Peter Rawlins and Maggie Hartnett

538

37

An Emerging and Exciting Future for Mixed Methods Research Design: Handbook Conclusions Cheryl N. Poth

542

Index

469 472

485

496

549

List of Figures 1.1 An overview of the Handbook by key numbers 1.2 Geographical locations of Handbook contributors on a 1954 Buckminster Fuller Airocean projection of the world map 1.3 Overview of Handbook chapters and unique contributions of sections 1.4 Key events, decisions and people influencing the development of this Handbook 2.1 A simplified presentation of the interconnection of four core components of mixed methods research 2.2 Evolution of diagrams used to depict the convergent mixed methods design type over time 4.1 Multi-stage transformative mixed methods design 5.1 The conceptual map of the notion “design” as used in the context of (mixed methods) research 5.2 The historical key influences on the development of mixed methods designs 6.1 Conceptual map of discourse development 6.2 Map of therapeutic relationship 8.1 Reflecting the fit of design components 8.2 Design flow of an explanatory sequential design 8.3 Design flow of an explanatory sequential mixed methods study 8.4 A methods-inference map of McCrudden and McTigue (2019) 8.5 Design components at the level of the whole study in Figure 8.4 8.6 Research questions at the level of the whole study and the level of the research strands in Figure 8.4 8.7 Research strand 1 in Figure 8.4 8.8 Research strand 2 in Figure 8.4 8.9 Developing the meta-inference in Figure 8.4 9.1 Using a tree metaphor as a sampling meta-framework for enhancing representation in mixed methods research 9.2 A flowchart illustrating the decisions in the tree sampling meta-framework 11.1 Big data components relevant for MMR researchers 12.1 The figure conceptualizes an integrated methodology as an example of intentional mixing at many levels. It is only a partial list of the ways that integration could occur when another methodology is paired with mixed methods. 12.2 Taking culture seriously in community mental health data analysis progression 12.3 Conceptualizing community-focused mental health 14.1 Procedural diagram of using a mixed method to create an indigenous cultural values instrument 14.2 The Delphi process 14.3 Correspondence analysis output for Malays’ cultural values scales and demographic categories 15.1 A braided river 15.2 The IBRLA framework and corresponding kaupapa M¯aori research principles 15.3 Ethnic groups of respondents in COVID assessment research compared with university figures 17.1 Participatory methods as part of an RCT of a complex intervention 18.1 Evolution of study design for illustrative case 19.1 Illustration of a CS-MMR embedding an MM intervention design in a game-based learning investigation 19.2 Illustration of an MM-CSR explanatory sequential design 19.3 Application of case study mixed methods (CS-MMR) using an exploratory sequential design

2 3 6 11 25 28 51 62 64 84 85 115 117 118 120 121 122 123 125 126 132 133 156 167 173 174 207 208 214 219 224 228 249 262 272 273 274

xiv

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

20.1 Guidelines related to item construction 20.2 Interview schedule development considerations 20.3 Examples of instrument development designs in mixed methods research 22.1 Crosstab created with MAXQDA Stats with using a code as categorial variable 22.2 Selecting categories (codes) and variables in MAXQDA Stats for statistical analysis 22.3 ANOVA table created with MAXQDA Stats using a code as the factor with three levels 22.4 Document Map in MAXQDA (partial view). The cases (interview respondents) are shown in a two-dimensional display using multi-dimensional scaling on codes as categorial variables and standardized variables 22.5 Mixed Methods Crosstab in MAXQDA. Multiple categorial variables (subcategories) form the rows, the values of a standardized variable form the columns 22.6 Case comparison table with mixed data in MAXQDA (partial view) 22.7 The QTT workspace of MAXQDA 23.1 Framework and iterative process of GTxA 23.2 Correspondence analysis of challenges in practising mixed methods research 23.3 Context of the phrase “research question” 23.4 The context for the keyword “timing” 23.5 A screenshot of qualitative data analysis with MAXQDA 23.6 Three hierarchical code groups of hurdles in mixed methods nursing research 23.7 An example of auto-coding rules for KH Coder 23.8 Results of auto-coding using KH Coder 23.9 Integrated results of grounded theory analysis and auto-coding 24.1 The initial hybrid design (two planned phases and an optional third phase) 24.2 The final hybrid design (three phases) 24.3 The integrated sampling strategy to gain access to key populations 24.4 Joint display: integration through stepwise data collection within and across phases 24.5 Screen capture of the gameplay from one screen recording 24.6 Joint display: integrated data analysis within and across phases 24.7 Percentage of participants (n = 34) who reported using English 24.8 A written, step-by-step procedure to protect the third party 25.1 Critical race mixed methodology with secondary data (CRMM+SD) 26.1 The Web 2.0 for Teachers of Spanish program and the TPACK model 26.2 Procedural diagram 26.3 TPACK-based joint display 26.4 Participants’ demographic data 26.5 Demographic data examination in case-oriented merged analysis 26.6 Case-oriented merged analysis visuals unfolding from joint display 26.7 Case-oriented merged analysis visual display S5.1 Summary of the ways a cultural component impacts research design among chapter authors 28.1 Procedural diagram of the convergent study design 28.2 ki-shyou-ten-ketsu style and the four-stage framework in clinical dialogue 30.1 Distribution of the five provinces in China 30.2 The team consists of global and local research members 30.3 Key functions of process evaluation and relations among them 30.4 Scaffolded mixed methods design integrated mixed methods interventional approach with a framework of developing and evaluating complex interventions 30.5 Implementation matrix focused on culture diversity based on scaffolded mixed methods

281 283 285 319 319 320 320 321 322 324 331 334 335 336 337 338 338 339 340 351 352 353 353 354 354 355 356 363 374 375 376 379 380 381 382 395 415 417 436 437 438 439 439

List of Figures

31.1 Data types and relation with primary parameters (frequency, order, and duration), according to Bakeman (1978) 31.2 Graphic information about the psychotherapeutic intervention performed in Carrilet Training and Research Center 32.1 Jannet’s research diagram based on Patel et al. (2016) 32.2 Brandi’s research diagram included in her mock mixed methods research design proposal final 32.3 Karis’s merging integration joint display based on Olkin et al. (2019) 32.4 Sasha’s building integration joint display for adding a quantitative phase to Robinson-Wood et al. (2015) 34.1 Levels of intersection in a mixed methods translational research study 34.2 Mixed methods translational research process 34.3 Mixed methods translational research framework 36.1 Key contributors to our mapping process and outcomes

xv

452 453 477 478 479 480 500 501 502 528

List of Tables S1.1 Summary of Section 1 chapters: Inspiring Diversity and Innovation in Mixed Methods Design 2.1 Intent, procedures and wording for mixed methods integration 5.1 The varied use of the notion “design” within the context of mixed methods research 5.2 Examples of early classifications of studies by their ways of using/combining quantitative and qualitative approaches (adapted from Niglas, 2004) 5.3 Selected most recent typologies of mixed methods designs S2.1 Summary of Section 2 chapters: The Craft of Mixed Methods Research Design 7.1 Synthesis of the practice of embracing emergence in mixed methods research 8.1 Linking research questions, theoretical basis, data sources and methods of data analysis 8.2 Design elements shown and not shown by each of the four visualizations 11.1 MMR studies using big data in Scopus until 2021 (n = 6) 11.2 Consideration of big data components 11.3 Consideration of ethical principles for big data 12.1 A multilevel framework to identify the unique contribution of each method and the potential synergies of MM-GTM, organized by philosophical assumptions and research procedures 12.2 Visualizing the analysis in a generic MM-GTM design with an abductive component and an iterative loop 12.3 Deconstructing the levels of mixing in Westhues et al. (2008) with the multilevel framework S3.1 Summary of Section 3 chapters: Expanding Mixed Methods Design Approaches 13.1 Implications of Cole’s three questions for each stage of the research process 13.2 Cigarette smoking in African-Caribbean young women: flowchart of the basic procedures in implementing a convergent design 14.1 Importance rate and expert consensus 14.2 Malay values and their dimensions 14.3 The experts’ final evaluation results 14.4 Final Malay value items 14.5 Respondents’ demographic profile 14.6 Results of the field test 15.1 Three approaches to research with M¯aori 17.1 Approaches to evaluating interventions by research paradigms 18.1 Models for mixed methods phenomenological research 19.1 A comparison of key attributes of MMR and CSR and the commonality between MMR and CSR 20.1 An example of a questionnaire framework for quantitative instruments 20.2 Interview schedule framework 20.3 Specifications for quantitative instruments aligned with qualitative data 20.4 Specifications for qualitative instruments aligned with quantitative data 21.1 Key MM-GT article citations in an MM-GT review pool 21.2 Best practices for grounded theory features to include in published MM-GT studies 21.3 Best practices for mixed methods features to include in published MM-GT studies 21.4 Correlations between methodological citations and mixed methods-grounded theory best practices

18 31 61 65 67 98 105 116 119 159 160 161 170 171 172 188 194 198 209 210 211 212 213 213 222 247 259 271 280 283 286 286 295 296 296 297

List of Tables

21.5 Theories, frameworks, models or typologies developed in MM-GT studies S4.1 Summary of Section 4 chapters: Designing Innovative Integrations with Technology 23.1 Key terminologies and definitions 23.2 Examples of focused coding (mentoring) 24.1 Ten strategies for innovative mixed methods integrations: valuing diverse voices and perspectives across contexts 25.1 Common challenges and solutions when integrating secondary data from ERM groups in mixed methods research 26.1 Mean and individual differences for pedagogy-related scales 26.2 Mean and individual differences for technology-related scales 26.3 Mean and individual differences for content-related scales S5.1 Summary of Section 5 chapters: Navigating Research Cultures in Mixed Methods Design 29.1 Key terms and definitions 30.1 The diet and culture in five provinces in China 30.2 The health belief model (HBM) and derived intervention used in the example project 30.3 Joint display of fidelity, delivered, reach, receipt and meta inferences by each component of intervention 31.1 Software designed from the framework of the Spanish research culture 31.2 Basic quantitative data analysis techniques in systematic observation studies (with methodological contributions made from the Spanish research culture) 31.3 Fragment of the observation instrument (field format combined with category systems) used in psychotherapy and elaborated ad hoc. This fragment includes a small part of the categories of the therapist 31.4 Record fragment where the initial transcription is shown, barely systematized and the coding to which it gives rise, thanks to the observation instrument. The + sign indicates co-occurrence, both for the therapist and for the patient 31.5 Table of parameters corresponding to the analysis of polar coordinate, with VSUP as focal behaviour and all the others as conditional. Only the conditioned behaviours that generate significant (*) and very significant (**) vectors have been selected 31.6 Fragment of the observation instrument (field format modality) elaborated ad hoc S6.1 Summary of Section 6 chapters: Exploring design possibilities and challenges for mixed methods research 33.1 Eight principles for guiding responsible research in the field of mixed methods research 36.1 Key characteristics of previous mixed methods research handbooks 36.2 This Handbook’s six section titles, leads, key questions and unique issues 36.3 Mapping projections from Mertens et al. (2016b), Creswell’s (2022) symposium themes, topics, and example chapters from this Handbook 36.4 Mapping Handbook chapters onto the evolving directions for mixed methods research design

xvii

299 312 333 337 350 364 378 378 378 394 423 436 440 442 448 451 454 455

455 456 470 487 530 531 533 534

List of Box 6.1 Key definitions

80

Notes on the Editor, Section Leads, and Chapter Contributors THE EDITOR Cheryl N. Poth is a Professor in the Centre for Research and Applied Measurement and Evaluation in the Faculty of Education at the University of Alberta (Canada) and an award-winning instructor and author. Her teaching and research interests have been related to enhancing research quality, complexity-informed study design, and collaborative research teams in the fields of education and the health sciences. She teaches mixed methods research courses and workshops around the globe and served as the fourth president of the Mixed Methods International Research Association. She serves as associate editor of the impactful Journal of Mixed Methods Research (Sage) and as an editorial board member of several journals. She has guest edited several mixed methods-focused special issues, including in the International Journal of Qualitative Inquiry, Journal of Mixed Methods Research, and the Canadian Journal of Program Evaluation, and authored works across a wide variety of Educational, Evaluation, Research Methods, Medical, and Health Sciences journals, encyclopedias, and books. She has served in several past leadership roles, including as Associate Dean (Research), Associate Director (Assessment) at the Centre of Teaching and Learning, Associate Chair (Undergraduate) of the Department of Educational Psychology, and Measurement, Evaluation, and Data Science Program Coordinator. Dr. Poth has an adjunct appointment in the Faculty of Medicine and Dentistry, and serves as the methodologist on several cross-disciplinary research teams. Her books, including the 4th edition of the Qualitative Inquiry & Research Design with John W. Creswell (2017, Sage), Innovation in Mixed Methods Research: Guiding Practices for Integrative Thinking with Complexity (2018, Sage), and Research Ethics (2021, Sage), are inspired by the practice dilemmas experienced in the field. Together with Peggy Shannon-Baker, she was awarded the 2023 American Educational Research Association Division D Significant Contributions to Research Methodology (Mixed Methodologies).

THE SECTION LEADS Elizabeth G. Creamer is a Professor Emerita in Educational Research and Evaluation from Virginia Tech University in the US where she taught several graduate level research methods courses in mixed methods and grounded theory. She is the author of two textbooks promoting an integrated approach to mixed methods, An Introduction to Fully Integrated Mixed Methods Research (2018) and Advancing Grounded Theory with Mixed Methods (2021). She is editor-in-chief of the mixed methods section of a methodological journal, Methods in Psychology, and served as co-editor of three special issues for journals on partnering mixed methods with qualitative research. Jessica T. DeCuir-Gunby is a Professor of educational psychology at the University of Southern California’s Rossier School of Education. She is also an associate editor of the Review of Educational Research journal. Her research interests include the impact of race and racism on the educational experiences of African American students, emotions and coping related to racism, Critical Race Theory, and mixed methods research. She loves cooking, traveling, and spending time with her husband, son, and two cats. Elsa Lucia Escalante-Barrios is an Associate Professor at the Universidad del Norte in Colombia where she teaches mixed methods courses for graduate students. Dr. Escalante-Barrios’s research areas include early childhood education, child development, cross-cultural research methods, and mixed methods. Dr. Escalante completed her PhD in Human Sciences at the University of Nebraska-Lincoln,

xx

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

where she completed the Mixed Methods Research Certificate. Her dissertation was awarded a Patrice Engle Dissertation Award for Global Early Child Development from the Society for Research in Child Development (SRCD) for a mixed methods cross-cultural multiple case study about feeding practices in the Latino population. She is currently the president of ALIMM (Latin American Mixed Methods Research Association). Dr. Escalante has published on the topics of childhood education and mixed methods research in several international journals, including Plos One, Research in Developmental Disabilities, Early Childhood Research Quarterly, and International Journal of Early Childhood. Sergi Fàbregues is an Associate Professor of Research Methods at the Department of Psychology and Education of the Universitat Oberta de Catalunya (Barcelona, Spain), Associate Editor of the Journal of Mixed Methods Research (JMMR) and PLOS One, and Associate Member of the University of Michigan Mixed Methods Program. His qualitative and mixed methods research interests include quality appraisal, integration, analysis, and the process of carrying out systematic methodological reviews. He has led several workshops internationally. He has worked collaboratively on funded research projects in various fields on a number of research topics, including nutrition, child sexual abuse, children and adolescents with developmental disabilities, gender and Information and Communications Technology (ICTs), and ICTs in education. In 2021, he co-edited the JMMR Virtual Special Issue on “Quality in Mixed Methods Research”. Timothy C. Guetterman is an interdisciplinary, applied research methodologist specialized in mixed methods research at the University of Michigan. His methodological goal is to advance rigorous methods of mixed methods research, particularly strategies for integrating qualitative and quantitative research. He applies mixed methods research to investigate informatics technology to improve health services, communication, and simulation training. Tim is also actively engaged in developing research methods capacity as an investigator for foundation grants and the NIH Mixed Methods Research Training Program for the Health Sciences. He serves as Co-Editor-in-Chief for the Journal of Mixed Methods Research. He coauthored the sixth edition of Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research with John W. Creswell. Maggie Hartnett is an Associate Professor and Director of the Teaching Academy responsible for academic staff professional development at Massey University, New Zealand. She is a Senior Fellow of AdvanceHE and has worked in the tertiary education sector for over 20 years. She is a mixed methods researcher and has served as an executive board member for the Flexible Learning Association of New Zealand (FLANZ) and as the Associate Editor for the Journal of Open, Flexible and Distance learning (JoFDL). She is a current board member of Ako Aotearoa, a government-funded organization that supports New Zealand’s tertiary sector. Sophia L. Johnson brings experiences as a mixed methods research instructor, editorial board member of the International Journal of Multiple Research Approaches and as a School of Pharmacy faculty member at the University of Maryland, Baltimore to her role as section co-lead. Her research is at the intersection of population health and chronic disease pharmacotherapy, focusing on understanding chronic disease self-management and related barriers and facilitators, particularly for patients with complicated drug therapy regimens from low-resourced communities. With co-author Nataliya Ivankova, she contributed a chapter “Designing Integrated Mixed Methods Action Research Studies” to the 2022 Routledge Handbook for Advancing Integration in Mixed Methods Research among other publications. José F. Molina-Azorin is a Professor at the Department of Management in the University of Alicante (Spain). His main research topics are strategic management, organizational design, environmental management, quality management and sustainability in the tourism industry. From a methodological perspective, he is interested in mixed methods and multilevel research. He uses mixed methods research in his substantive studies, and he has also conducted systematic reviews and prevalence studies about the use of mixed methods in multiple management areas. He served on the Mixed Methods International Research Association’s Presidential Task Force with a report about the Future of Mixed Methods. He is Co-Editorin-Chief of the Journal of Mixed Methods Research. He has been guest co-editor of two special issues on mixed methods in business and management published in the International Journal of Multiple Research Approaches and Organizational Research Methods.

Notes on the Editor, Section Leads, and Chapter Contributors

xxi

Peter Rawlins is an Associate Professor and the Director of Academic Quality for the College of Humanities and Social Science at Massey University, New Zealand. Before joining the university, he was a secondary mathematics teacher for 18 years. Peter was a founding member of the Mixed Methods International Research Association Oceania Regional Chapter (MMIRA-ORC) and was the conference chair for the inaugural MMIRA-ORC conference held in Wellington, New Zealand, in 2019. He is currently Secretary for the MMIRA-ORC committee. Peter was also the chair of the conference committee for the MMIRA global governance board and led the planning group for the online global MMIRA conference held in August 2022. Judith Schoonenboom holds the chair for empirical pedagogy at the University of Vienna, Austria. She specializes in mixed methods research in education. Her research interests include mixed methods research design and the foundations of mixed methods research. Judith has been an associate editor of the Journal of Mixed Methods Research since 2014. In 2018, she was co-organizer of the biannual global conference of the Mixed Methods International Research Association (MMIRA), which was held in Vienna. She participated in the committee that prepared the 2022 online global MMIRA conference. Judith has served as MMIRA’s President-Elect, President, and Past-President in the years 2019–2022. Peggy Shannon-Baker is an associate professor of educational research at Georgia Southern University, where they teach and research mixed methods and other forms of research in primary, secondary, postsecondary, and teacher education. Peggy is also an associate editor of the Journal of Mixed Methods Research. They come to mixed methods especially interested in looking for the connections between data, theory, and practices because they are at heart a lumper (someone who primarily sees similarities) and not a splitter (someone who primarily separates things into categories).

THE CHAPTER CONTRIBUTORS Annika Agni is an undergraduate student at the University of Michigan–Ann Arbor. She is currently a Research Assistant for the Department of Family Medicine, and works on multiple projects in the health science field that involve mixed methods research, including a study on hypoglycemia self-management in adults with type 1 diabetes, and a scoping review on mixed methods articles that focus on racial/ethnic disparities across the cancer care continuum. She first became involved with research through the Undergraduate Research Opportunity Program at the University of Michigan, and her research interests include public health and health science topics. Jami L. Anderson is a postdoctoral research associate with the Implementation Science Center for Cancer Control at Washington University in St. Louis. Dr. Anderson’s research focuses on the intersection of mixed methods research and knowledge translation for cancer prevention and control in rural and remote areas. She is also an advocate for the development of innovative rapid-cycle implementation research methods and learning health systems to support evidence-based policies and shift research into practice at organization, state, and national levels. Dr. Anderson is a member of the Mixed Methods International Research Association, Academy Health, and the American Society of Clinical Oncologists. M. Teresa Anguera holds degrees in Psychology and Law. She is an Emeritus Professor at the Faculty of Psychology from the University of Barcelona. Since 1972 she has taught in various methodological subjects, and over four decades she has been working on observational methodology. In the last two decades, she has dedicated herself to the study of mixed methods and the positioning of systematic observation from this approach. She has coordinated numerous competitive nationally funded research projects. She has directed/co-directed 68 doctoral theses. She has numerous publications at a national and international level, and she has been part of research evaluation committees in several countries. She has carried out research stays in Chile, México, Portugal, and the USA. Her h index is 70. She has had management responsibilities at the University of Barcelona, including serving as Vice-rector for Scientific Policy and Vice-rector for Teaching and Scientific Policy.

xxii

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Mandy M. Archibald is an Assistant Professor in the College of Nursing, University of Manitoba, Research Scientist at the Children’s Hospital Health Research Institute of Manitoba, an interdisciplinary artist, and Senior Research Fellow (Academic Status) at Flinders University, South Australia. She completed a three-year postdoctoral fellowship in a Transdisciplinary Centre of Research Excellence where she studied transdisciplinary research processes. Mandy is also an associate editor of the Journal of Mixed Methods Research and the International Journal of Multiple Research Approaches. As an applied research methodologist, she leads a research program integrating diverse bodies of knowledge generated through arts-based, mixed-methods, co-designed, and transdisciplinary research in chronic illness contexts. In the field of mixed methods, this work has advanced integration of arts-based and mixed-methods research mergers, qualitatively driven mixed-methods research, investigator triangulation and transdisciplinary research for impact, and has published in leading journals such as the Journal of Mixed Methods Research, International Journal of Qualitative Methods, American Behavioural Scientist, as well as mixed methods chapters for publishers Springer and Routledge. Eulàlia Arias-Pujol is an Associate Professor of “Evaluation and psychopathology in Childhood and Adolescence” at the Faculty of Psychology and Educational Sciences and Sports Blanquerna of the Ramon Llull University (URL). She is a specialist in clinical psychology with extensive training in psychoanalysis. She teaches various seminars on professionalizing skills for the Degree in Psychology. She also collaborates in postgraduate courses and in the doctoral program in psychology. She has directed doctoral theses from the Research Group on Couple and Family of the URL and has published several scientific articles in national and international journals. Her lines of research follow the mixed methods methodology and focus on studying the interaction in individual and group psychotherapeutic interventions. She has received several awards for research on childhood autism carried out in collaboration with the Carrilet Training and Research Center of Barcelona. Wayne A. Babchuk is a Professor of Practice in the Qualitative, Quantitative, and Psychometric Methods (QQPM) program in the Department of Educational Psychology at the University of Nebraska–Lincoln (UNL) and holds courtesy appointments in the Departments of Anthropology and Sociology. He is an applied research methodologist focusing on the history, epistemology, application and instruction of qualitative and mixed research across disciplines, research ethics, grounded theory, ethnography, grounded ethnography, mixed methods-grounded theory, and community-based participatory research. He is cofounder and co-chair of UNL’s Qualitative and Mixed Methods Group and serves as Editor-in-Chief for the Journal of Ethnographic and Qualitative Research (JEQR). Dr. Babchuk has facilitated workshops in qualitative and mixed methods research for the African Doctoral Academy (ADA), Stellenbosch University, Stellenbosch, South Africa, and in Osaka, Japan, for the Japan Society for Mixed Methods Research affiliated through the University of Michigan Mixed Methods Program, as well as in the United States. Maya Bartel received her doctorate in HDFS at Iowa State University in 2022. Maya also earned a Graduate Certificate in Developmental and Family Sciences Advanced Research Design and Methods in 2019, and an MS in HDFS in 2021 from ISU. Maya’s research interests include adoption, qualitative research methodologies, and gender and racial identity development. Maya has contributed to multiple research efforts spanning parenting education, substance abuse, data systems, child welfare, mathematics education, and mixed methods. In recognition of her excellence as an emerging scholar and community servant, she received the 2021 HDFS Research Excellence Award, served as the Graduate Student Marshal in May 2021, earned the 2022 Martin Luther King, Jr. Advancing One Community Award, and was a 2022 Woman of Achievement. In 2016, she graduated magna cum laude from Portland State University with a BS in community health education and women’s studies. Ayesha S. Boyce is an Associate Professor in the Division of Educational Leadership and Innovation at Arizona State University. Dr. Boyce’s scholarship focuses on attending to value stances and issues related to diversity, equity, inclusion, access, cultural responsiveness, and social justice within evaluation—especially multi-site, STEM, and contexts with historically and systematically marginalized populations. She also examines teaching, mentoring, and learning in evaluation. Dr. Boyce is a 2019 American Evaluation Association Marcia Guttentag Promising New Evaluator Award recipient and a 2019 UNC Greensboro School of Education Distinguished Research Scholar Award recipient.

Notes on the Editor, Section Leads, and Chapter Contributors

xxiii

Lisbeth M. Brevik is an award-winning teacher educator and researcher. She joined the University of Oslo, Norway, as a full Professor in 2021 within the Faculty of Educational Sciences and was appointed Academic Chair of the European University Alliance Circle U. She has developed and taught mixed methods and research design courses, in addition to supervising doctoral and Master’s students. Her field of research is English language education, and, over the years, she has gained expertise in game-based mixed methods research integrations. She is the PI of several externally funded research projects. In 2018, she was awarded membership of Kellogg College, University of Oxford, UK, and in 2020, she was recognized with the University of Oslo’s Education Prize and Teaching Excellence Award. She is the author of articles, chapters, and anthologies, and currently is an editorial board member of the International Journal of Multiple Research Approaches (IJMRA) and Reading in a Foreign Language (RFL). Carolina Bustamante has been an Associate Professor in the School of Education at the State University of New York at Old Westbury since 2014. She coordinates the Graduate Spanish Adolescence Education Program and teaches courses on language pedagogy for second-language learners and heritage speakers, second-language acquisition, and Spanish. She has also worked on seminars on mixed methods research at the TESOL division at the University of Cambridge, England, and Graduate School of Education at the University of Pretoria, South Africa. Her published research has focused on professional development on technology for teachers of Chinese, German, and Spanish, development of heritage speakers of Spanish as language teachers, integrated performance assessment in Spanish classrooms, and mixed methods research, particularly innovative ways to mix qualitative and quantitative data. Philippa Butler is a lecturer of research methods in the Institute of Education, Massey University, New Zealand. She has also worked as a research officer and is very experienced at conducting externally funded education research projects. Her teaching interests are in mixed methods research and research methodology, and her research interests include ethnic group identifications and issues of equity in education. Jane Cahill is a lecturer in the School of Healthcare, University of Leeds. Jane has published widely in the field of psychological therapy effectiveness research, as well as continuing to work and publish within the wider field of mental health. Her research has focused on the therapeutic alliance, clinically representative approaches to psychotherapy research, and mental health and workforce mental health issues. Jane’s research interests have their origin in psychotherapy research, and her career as a researcher began with working on research programs focused on the effectiveness of psychotherapy in clinically representative settings, working with a range of clinical datasets and addressing clinical, research and practice questions. Key areas of interest and expertise have been processes and outcomes in clinically representative research, methodology of clinically representative research, measure development and the developments of the wider methodological paradigm of clinically representative research. Roslyn Cameron is a Professor and Director of the Centre for Organisational Change and Agility (COCA) at Torrens University Australia, Australia. She is Co-Convenor and founder of the Mixed Methods Research Special Interest Group of the Australian and New Zealand Academy of Management (ANZAM), a previous Board Member of the Mixed Methods International Research Association (MMIRA), and a member of the Australian Human Resources Institute (AHRI) Advisory Research Panel in Australia. She has been the recipient of several large-scale workforce development research grants and an array of smaller scale research grants related to skilled migration, work readiness/employability and future skilling/future of work for the 4th Industrial Revolution, totalling $AUD1.9m, and has over 90 publications. P. Paul Chandanabhumma is an Assistant Professor in the Department of Family Medicine, and Executive Committee Member of the Michigan Mixed Methods Program. He completed his PhD in Community Health Sciences at UCLA Fielding School of Public Health. His mixed methods dissertation research examined the influence of group diversity on the achievements of community-based participatory research partnerships. His research interests include health inequities, race, culture, community engagement, and the social production of medical and public health practices. Hongling Chu is an Assistant Professor working in Research Center of Clinical Epidemiology, Peking University Third Hospital, Beijing, China. She is a visiting scholar for the MMR Project Group at the University of Michigan. Dr. Chu led the establishment of the MMIRA—China Chapter (MMIRA–ChC)

xxiv

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

in 2020 and the translation for A Practical Guide to Using Qualitative Research with Randomized Controlled Trials and The Mixed Methods Research Workbook: Activities for Designing, Implementing, and Publishing Projects. From 2018 to 2022, she organized the MMR conferences and seminars. Her areas of expertise are in implementation science, mixed methods, qualitative research and clinical trials. She participated in three large-scale multicenter intervention research projects in the design, implementation and process evaluation, and published several related articles. Loraine D. Cook is a Professor at the School of Education, University of the West Indies. She lectures in Research Methods and Educational Psychology Mona. She has authored and co-authored peer-reviewed book chapters and journal articles. Dr. Cook is the founding president of the Mixed Methods International Research Association–Caribbean Chapter (MMIRA–CC). She is a founding co-chief editor of the Caribbean Journal of Mixed Methods Research (CJMMR). Dr. Cook was the recipient of the Fulbright Visiting Researcher Award (2011). Dr. Cook has been a visiting scholar in Applied Psychology at New York University (NYU) and the Faculty of Education, Language and Literacy Education, University of British Columbia, Canada. Julie A. Corrigan is an Assistant Professor at Concordia University in Montreal where she teaches courses in educational technology and research methods. Her Canadian and Quebec funded research is focused on closing the socio-digital divide by equipping teachers with the practices and resources to teach and assess digital literacies. Along with co-author Anthony Onwuegbuzie, she has delivered a number of presentations in the area of mixed methods research, including a workshop for Canadian researchers on using mixed research to achieve more ethical and rigorous research. Dr. Corrigan has also co-authored methodological and theoretical papers on mixed research, including those on sampling (The Qualitative Report), mixed research in special education (Research in the Schools), and mixed research in human resources and development (Human Resource Development Quarterly). She runs research writing support groups at Concordia University where she enjoys challenging graduate students to consider their sampling designs as a means to conduct more transparent, rigorous, ethical, and equitable research. Elizabeth G. Creamer is Professor Emerita from the educational research program in the School of Education at Virginia Tech in the US, where she taught doctoral-level research methods courses in qualitative research and mixed methods for more than fifteen years. A prolific writer, Dr. Creamer is the author of two recent books: An Introduction to Fully Integrated Research (2018) and Advancing Grounded Theory with Mixed Methods Research (2021). Her research focus is on integration in qualitatively oriented mixed methods research. A third textbook is in progress: Leveraging Visual Displays in Mixed Method Research. John W. Creswell is a Senior Research Scientist in the Michigan Mixed Methods Program, University of Michigan, Ann Arbor, USA. He has authored numerous articles and 33 books on mixed methods research, qualitative research, and research design. He has held an Endowed Chair at Nebraska, co-founded the Journal of Mixed Methods Research, co-led the NIH “best practices in mixed methods” study group, and lectured at Harvard University. He served as a Senior Fulbright Scholar in South Africa and in Thailand, was the founding President of the Mixed Methods International Research Association, co-directed the University of Michigan Mixed Methods Research Program, and co-authored “standards” on mixed methods research for the American Psychological Association. He makes his home in Honolulu, Hawaii, and Ashiya, Japan. Shanlee Davis is a pediatric endocrinologist and the Director of the eXtraOrdinary Kids multidisciplinary Turner Syndrome Clinic at Children’s Hospital Colorado, and Assistant Professor of Pediatrics at the University of Colorado School of Medicine. She has content expertise in sex chromosome aneuploidies and other genetic conditions associated with endocrinopathies. Dr. Davis’ clinical and translational research program incorporates interdisciplinary methods and perspectives to answer clinically important research questions with the overarching goal of improving health and wellbeing for these individuals. ​​ Manuela De Allegri has a diverse academic training in Sociology, Health Economics, and Public Health. She leads the Research Group in Health Economics and Financing at the Heidelberg Institute of Global Health. Through the systematic application of mixed methods approaches, Manuela combines quantitative

Notes on the Editor, Section Leads, and Chapter Contributors

xxv

and qualitative methods of data collection and analysis to her work in Global Health. Her work focuses primarily on the evaluation of health financing and system reforms in low- and middle-income countries. Melissa DeJonckheere is an Assistant Professor with the Department of Family Medicine and Executive Committee Member of the Michigan Mixed Methods Program. She is a health services researcher and mixed methodologist specializing in qualitative and participatory research design. Her research focuses on understanding the psychological and social contexts of chronic disease management in adolescence. She previously was a postdoctoral research fellow in the Michigan Mixed Methods Program. Dr. DeJonckheere completed her PhD in Educational Studies at the University of Cincinnati, where she worked on qualitative and mixed methods projects in both the education and health fields. Luca Del Giacco is a clinical-dynamic psychologist born in Perugia (Italy). He obtained the Master’s degree cum laude in 2014 at the University of Padua (Italy) and collaborated with its clinical service until 2017. From 2017 to 2020, he collaborated in several research projects on communication in psychotherapy at the University of Padua; furthermore, in 2018, he was a member of the work team in the project “Physical activity and sport as a healthy lifestyle enhancer: evaluation of sports behavior from non-intrusive methodologies” at the University of Barcelona. He is the author of various publications on therapeutic discourse and has participated in several national and international congresses. Currently, he is a member of the work team in the project “New approach to research in physical activity and sport from a mixed-methods perspective” at the University of Barcelona, and is a reviewer for the Research in Psychotherapy: Psychopathology, Process, and Outcome. Jenny Douglas is a Senior Lecturer in Health Promotion in the Faculty of Wellbeing, Education and Language Studies at the Open University. The focus of her research and activism is intersectionality – exploring how ‘race’, class and gender affect particular aspects of African Caribbean women’s health. She established and chairs the Black Women’s Health and Wellbeing Research Network (www.open.ac.uk/black-womens-health-and-wellbeing). She is a medical sociologist with a PhD in Women’s Studies from the University of York, the focus of which was cigarette smoking and African Caribbean young women in the UK. This study employed intersectionality-informed mixed methods research. She was a visiting scholar at George Washington University, Washington, DC during the tenure of a National Centre for Research Methods Fellowship on intersectionality informed research methods. She is a Plumer Visiting Research Fellow at St. Anne’s College, University of Oxford. Cherie D. Edwards is an Assistant Professor at Virginia Commonwealth University’s School of Medicine. Dr. Edwards earned her PhD in Educational Evaluation and Research from Virginia Tech. Her research interests are in exploring the implementation of mixed methods and visual research designs in studies examining social justice and equity in education research. Michael D. Fetters is Professor, Department of Family Medicine, Director, the Mixed Methods Program, and Director, Japanese Family Health Program at the University of Michigan. He has served as Co-Editorin-Chief of the Journal of Mixed Methods Research and co-edited a special issue on conducting research in primary care in Family Medicine and Community Health (2019). His research focuses on cultural influences on medical decision-making, health services research, and qualitative and mixed methods research methodology. He authored The Mixed Methods Research Workbook: Activities for Designing, Implementing, and Publishing Projects (2020), Sage. Dawn Freshwater is Vice-Chancellor of the University of Auckland, and Clinical Professor of Mental Health at the University of Western Australia. She became the University’s first female Vice-Chancellor in March 2020, after serving as the University of Western Australia’s Vice-Chancellor and Senior Deputy Vice-Chancellor and Registrar for six years (2014–2020). Professor Freshwater was the first female Chair of the G08 Research Intensive Universities in Australia and is currently Chair of UNZ Research Committee, a board Director of Research Australia, and a Steering Committee Member for the Asia Pacific Rim of Universities. Professor Freshwater is a highly experienced and driven supporter of translational research, methodological innovations for health research, and research-led teaching. Her contribution to the fields of public health and in researching leadership practices won her the highest honor in her field – the Fellowship of the Royal College of Nursing (FRCN). She has contributed to almost 200 publications

xxvi

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

and was recently elected MAE (Member of the Academia Europea). She previously served as Editor of the Journal of Mixed Methods Research. Timothy C. Guetterman is an interdisciplinary, applied research methodologist specialized in mixed methods research. He is faculty and Associate Director of the Mixed Methods Program at the University of Michigan. His methodological goal is to advance rigorous methods of mixed methods research, particularly strategies for integrating and intersecting qualitative and quantitative research. Funded by the National Institutes of Health (NIH), he investigates informatics technology to improve health services, communication, and simulation training. Tim is also actively engaged in developing research methods capacity as an investigator for foundation grants and the NIH Mixed Methods Research Training Program for the Health Sciences. He serves as Co-Editor-in-Chief for the Journal of Mixed Methods Research. Jori N. Hall has published numerous peer-reviewed works addressing issues of evaluation and research methodology, cultural responsiveness, and the role of values and privilege within the fields of education and health. She has authored an award-winning book, Focus Groups: Culturally Responsive Approaches for Qualitative Inquiry and Program Evaluation, and was selected as a Leader of Equitable Evaluation and Diversity (LEEAD) fellow by The Annie E. Casey Foundation. Dr. Hall is the 2020 recipient of the American Evaluation Association’s Multiethnic Issues in Evaluation Topical Interest Group Scholarly Leader Award for scholarship that has contributed to culturally responsive evaluation. Maggie Hartnett is an Associate Professor and Director – Teaching Academy at Massey University, New Zealand. Her research focuses on the intersection of technologies and pedagogies and their influence on learners’ and teachers’ experiences, motivation, engagement and behaviour in technology-enhanced, mediated and immersive learning contexts. Taichi Hatta is a lecturer at Shizuoka Graduate University of Public Health. His research interests lie in health science, medical sociology, medical ethics, cultural psychology, quantitative and qualitative research, and mixed methods. His doctoral paper, published online in 2018 in the Journal of Mixed Methods Research, demonstrated a new way of analyzing temporal changes of moods in physician–patient dialogues for cancer treatment at a hospital in Japan using a Japanese narrative framework. Following that, he was awarded a PhD from Kyoto University in 2019, and also received the Honorable Mention award for the Mixed Methods International Research Association’s Dissertation Award in 2022. He is one of the few scholars who has imported mixed methods to Japan, encouraging home students and researchers to engage in mixed methods research and writing articles in Japanese. Dr. Hatta translated Foundations of Mixed Methods Research (Tashakkori & Teddlie, 2009) into Japanese as one of the translation supervisors. Ivan I. Herbey is consultant and data analyst in the Schools of Medicine and Health Professions at the University of Alabama at Birmingham (UAB). As a trained epidemiologist and medical scientist with extensive expertise in qualitative and mixed methods research, he contributes to the design and implementation of funded research projects in social and health sciences, including patient-centered research and translational science. He is the lead consultant in the Mixed Methods and Qualitative Research and Evaluation (MMQRE) unit within the Center for Health Informatics for Patient Safety and Quality (CHIPS/Q) at UAB. Heinz Herrmann is Adjunct Professor at the Australian Graduate School of Leadership, and a Research Fellow and Senior Lecturer at Torrens University Australia. He is a member of scientific and editorial boards, and has a research focus on business, artificial intelligence, competitive bidding, applied ethics and mixed methods research. In industry, he is a CEO with more than 25 years’ standing in Technology, Media & Telecommunications (TMT), including commercial and NFP board director roles. Michelle C. Howell is an Assistant Professor at the University of Nebraska Medical Center where she serves as an Educational Researcher for the Interprofessional Academy of Educators, and teaches research methodology courses in the College of Public Health. Trained by John W. Creswell and mentored by Vicki Plano Clark, Dr. Howell has developed expertise in mixed methods research designs, with particular focus on instrument development procedures such as grounded theory analysis, cognitive interviews and factor analysis. Dr. Howell has served as an evaluator on numerous mixed methods projects funded by NIH, NSF, and IES, as well as private contracts.

Notes on the Editor, Section Leads, and Chapter Contributors

xxvii

​​ Mitsuyuki Inaba is a Professor at the College of Policy Science, Ritsumeikan University, Japan. His research interests include text mining, computational linguistics, and research methods. He has published several articles and research papers applying text mining and CAQDAS for forensic science. He is the president of the Japan Society for Mixed Methods Research. He is also a member of the Mixed Methods International Research Association, the Association for Computing Machinery, the Japanese Society of Artificial Intelligence and other academic associations. Nataliya V. Ivankova is Professor in the School of Health Professions at the University of Alabama at Birmingham. As an applied research methodologist working at the intersection of mixed methods, qualitative, community-based participatory, action and translational research, she directs an online graduate certificate program in Applications of Mixed Methods Research. She is a founding co-editor of the Mixed Methods Research Series (Sage), and is a qualitative and mixed methods research section editor for the 4th edition of the International Encyclopedia of Education (Elsevier). She also serves as an associate editor for the Journal of Mixed Methods Research. Natasha C. Johnson is an Assistant Professor of social work at Columbia University. She has experience using strengths-based methodologies, including mixed methods, person-centered quantitative methods, and qualitative methods, to address systemic inequities that hinder Black youths’ thriving. Dr. Johnson’s program of research examines critical developmental processes that facilitate resilient pathways to promote mental wellness and improve academic outcomes for Black youth. Additionally, she examines Black youth’s awareness of and responses to racial inequality, development of racial identity, and race-related experiences. Dr. Johnson earned her BA in psychology from Spelman College, and her MSW and PhD in Social Work & Personality Psychology from the University of Michigan. Tera R. Jordan is the Assistant Provost for faculty development and an Associate Professor of Human Development and Family Studies (HDFS) at Iowa State University (ISU). She has published scholarly work under the name Tera R. Hurt. Her research focuses on relationships and family wellbeing among underrepresented adults. She has taught advanced qualitative methods and mixed methods at the graduate level. An award-winning scholar, she has been honored by the ISU Department of Human Development and Family Studies, the ISU College of Human Sciences, and the University for her dedication and commitment to teaching, mentoring, community engagement, diversity enhancement, and inclusive excellence. Prior to her faculty appointment in 2012, she earned a dual-title PhD in HDFS and demography from The Pennsylvania State University in 2005, and worked as a research scientist at the University of Georgia from 2004 to 2012. Hisako Kakai is a Professor in international communications at Aoyama Gakuin University in Tokyo. Her Introduction to Mixed Methods Research is the first book regarding mixed methods research written by a Japanese scholar in Japanese. Dr. Kakai also translated Creswell’s A Concise Introduction to Mixed Methods Research and Charmaz’s Constructing Grounded Theory (1st edn). Since its inauguration, she has served as an editorial board member for the Journal of Mixed Methods Research. She also served as the founding president of the Japan Society for Mixed Methods Research (2015–2017). Vimala Judy Kamalodeen specializes in Math and Computer Science education at the School of Education, University of the West Indies, St. Augustine, Trinidad and Tobago. Her doctoral thesis used an eclectic mixed methods design focusing on hybrid data from educational online social networking. Vimala is the founding President of ITTPN Global, a professional learning network and focused on mixed methods research in game-based learning. Dr. Kamalodeen is a past president of the Caribbean Chapter of Mixed Methods International Research Association and successfully hosted the 3rd regional mixed methods conference in the Caribbean. Dae Hyun (Daniel) Kim currently serves as an Assistant Professor of Health Management and Policy at Georgetown University. He received his undergraduate degree in Psychology from the University of Michigan (2017) and doctorate degree in Health Services Administration from the University of Alabama at Birmingham (2020). His primary research interests focus on health literacy, health disparity, leadership competencies, strategic management and outcomes. Dr. Kim is an avid advocate of mixed methods research and is actively engaged in AUPHA, CAHME, AOM, and Academy Health.

xxviii

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Udo Kuckartz is a Professor Emeritus of Education and Research Methods at the Philipps University Marburg (Germany). He has authored 23 books and numerous articles on qualitative and quantitative methods, on computer-assisted qualitative data analysis, and on environmental issues. His books have been translated into several languages (English, Chinese, Japanese, and Spanish). He was one of the pioneers of computer-assisted analysis of qualitative data, earning his doctorate as early as the 1980s with a thesis on “Computer and verbal data: Opportunities for innovation of social science research techniques”. With this, he created the foundation for corresponding computer software (QDA software) that he developed in the following years (MAX, winMAX, MAXQDA). He has been principal investigator for projects and grants funded by the German Ministry of Environment and the German Environment Agency. His research focused on the topics of environmental attitudes and behaviour, sustainability, and perceptions of climate change. Spencer Lilley is an Associate Professor at the School of Information Management, Victoria University of Wellington. Spencer’s research interests focus on indigenous engagement with information, and the indigenisation of cultural heritage institutions. Spencer has tribal affiliations to Te Atiawa, Mua u¯ poko and Ng¯apuhi. Hueiming Liu is a public health physician and an conjoint senior lecturer with a PhD from the University of Sydney (awarded 2019) in addition to a BA, MBBS, and Master’s in International Public Health. She is a Senior Research Fellow, Centre Health Systems Science, and leads the process evaluation research program at the George Institute for Global Health. She brings expertise in mixed methods process evaluations, qualitative research, clinical trials, community engagement, program management, capacity building, and has worked extensively in cross cultural and interdisciplinary research. Julia Lohmann is an Assistant Professor at the London School of Hygiene and Tropical Medicine. She uses mixed methods in her research on health systems. Her work focuses more specifically on the health workforce and how it can be supported and strengthened in low- and middle-income countries. Jenevieve Mannell is a mixed methods researcher and expert in the use of qualitative methods alongside randomised controlled trials of complex interventions in global health. In 2018 she co-edited a special issue of Qualitative Health Research on innovative qualitative methods for RCTs, and has developed new methodological approaches including Visual Participatory Analysis (VPA) and Participatory Community Intervention Development (PCID). Her current research focus is the prevention of violence against women in high-prevalence settings, with ongoing projects in Afghanistan, India, Peru, Samoa, and South Africa. Joseph A. Maxwell is a Professor (Emeritus) in the Research Methods program in the College of Education and Human Development at George Mason University. His doctoral degree is in anthropology, but for the past 40 years his research and teaching have focused on methodology. He is the author of Qualitative Research Design: An Interactive Approach (3rd edn, 2013), and A Realist Approach for Qualitative Research (2012), as well as articles on qualitative and mixed methods research, indigenous North American societies, and medical education. His current research and writing deal with using qualitative and mixed methods for causal explanation and generalization; validity in qualitative and quantitative research; the history and breadth of mixed methods research; the value of philosophic realism for social research; and the importance of diversity and dialogue across research paradigms and methods. Joanne Mayoh is a Senior Academic at Bournemouth University who offers expertise in research methodology specializing in phenomenological research and mixed-methods research. She has a passion for methodology and has previously collaborated with internationally renowned authors on methodological journal articles that focus on the formal conceptualisation and review of Mixed Methods Phenomenological Research (MMPR). Joanne is proud to be a co-convenor of Bournemouth University Women’s Academic Network, and is a fierce advocate of equity, diversity, and inclusion. Cassandra McCall is an Assistant Professor in the Department of Engineering Education at Utah State University. Her research focuses on grounded theory applications to explore professional identity and disability identity formation in engineering. Dr. McCall has authored several publications that expand conceptions of grounded theory and promote its use in education research.

Notes on the Editor, Section Leads, and Chapter Contributors

xxix

Donna M. Mertens is Professor Emeritus at Gallaudet University with a specialization in mixed methods approaches that are designed to support social transformation. She has authored, co-authored, or edited many methodological books related to justice and human rights, most recently Mixed Methods Research, Program Evaluation Theory and Practice (2nd ed); Mixed Methods Design in Evaluation; Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods (5th ed); Indigenous Pathways into Social Research, and Transformative Research and Evaluation. She focuses on the intersection of research design and social, economic and environmental justice within the philosophical assumptions of the transformative paradigm. Dr. Mertens served as the Editor for the Journal of Mixed Methods Research, 2010–2014. She was President of the American Evaluation Association in 1998, and served on the board from 1997 to 2002; she was a founding board member of the Mixed Methods International Research Association. José F. Molina-Azorin is Professor at the Department of Management in the University of Alicante (Spain). His main research topics are strategic management, organizational design, environmental management, quality management and sustainability in the tourism industry. From a methodological perspective, he is interested in mixed methods and multilevel research. He uses mixed methods research in his substantive studies and he has also conducted systematic reviews and prevalence studies about the use of mixed methods in multiple management areas. He served on the Mixed Methods International Research Association’s Presidential Task Force with a report about the Future of Mixed Methods. He is Co-Editorin-Chief of the Journal of Mixed Methods Research. He has been co-guest editor of two special issues on mixed methods in business and management published in the International Journal of Multiple Research Approaches and Organizational Research Methods. Francisco Molinero is a doctor in Psychology. He is a researcher, mediator and psychotherapist. He is a specialist in Solution-Focused Brief Therapy and Motivational Interviewing. He has been formed in Microanalysis of the Therapeutic Dialog by the group of Janet Bavelas. He is an expert in qualitative research using digital tools such as ATLAS.ti (he is certified trainer) and ELAN. He incorporates observational methodology and mixed methods, the contributions of the Conversational Analysis and the methodology of Reflective Professional Practice with the aim of improving the effectiveness of the professional intervention in change processes. He is part of the Mixed Methods Research Group led by Dr. Anguera at the University of Barcelona. Katrin Niglas is an elected professor with the focus on research methods at Tallinn University (Estonia). Since 2011, she serves as a Vice-Rector for Research. After achieving the teacher training diploma and MA degree in Estonia, Katrin studied at the University of Cambridge and earned the MPhil degree in Educational Research in 1999. In her PhD dissertation, defended in 2004, she focused on the combined use of qualitative and quantitative methods in social and educational research. This topic continues to be her main research and writing interest. She is a member of the editorial board of the Journal of Mixed Methods Research, as well as of the International Journal of Multiple Research Approaches. She is also a co-author of a chapter published in the Sage Handbook of Mixed Methods Research (2nd edn), published in 2010. Anthony J. Onwuegbuzie is a Senior Research Associate at the University of Cambridge. Further, he is a Distinguished Visiting Professor at the University of Johannesburg; Professor Extraordinarius at the University of South Africa; Visiting Senior Scholar, St. John’s University, New York; and an Honorary Recognised Supervisor (online), University of Liverpool. As Past President of the Mixed Methods International Research Association (MMIRA), and as someone who is passionate about qualitative research, quantitative research, and mixed research, Tony co-authored the Sage mixed research textbook with Rebecca Frels (Lamar University), Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach. He is former editor of Educational Researcher. Currently, he is Editor-in-Chief of both the International Journal of Multiple Research Approaches and the Journal of Mixed Methods Studies. Vicki L. Plano Clark is Professor of Research Methods in the School of Education at the University of Cincinnati, USA. She is an applied research methodologist who studies, teaches, mentors, and writes about the adoption and use of mixed methods research. Her scholarship focuses on resolving methodological issues associated with mixed methods designs, and understanding larger contexts that influence the application of mixed methods research. Her writings include Designing and Conducting Mixed Methods Research (2018, co-authored with John W. Creswell) and Mixed Methods Research: A Guide to the Field

xxx

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

(2016, co-authored with Nataliya V. Ivankova). She currently serves as an Associate Editor for the Journal of Mixed Methods Research and is Co-Editor of the Mixed Methods Research Series with Sage. Audrey Prost trained in social anthropology, epidemiology and public health. She has used qualitative methods within cluster randomised controlled trials in global health for over a decade, with an emphasis on mixed methods approaches in process evaluation. Her current research focuses on developing and evaluating participatory interventions to improve women’s, children’s and adolescents’ health in Eastern India in partnership with the civil society organisation, Ekjut, and local government. Stefan Rädiker is a consultant and trainer for research methods and evaluation. His research focuses on the computer-assisted analysis of qualitative and mixed methods data. Stefan is a passionate lecturer and has given more than 200 workshops and webinars on analysing qualitative data, conducting mixed methods studies, and doing evaluation. His intention is to teach the application of research methods in a structured and easy-to-understand manner. In addition to several book chapters, he has co-authored and co-edited several books on qualitative and quantitative research methods, among others The Practise of Qualitative Data Analysis: Research Examples Using MAXQDA, Analyzing Qualitative Data with MAXQDA: Text, Audio, and Video, and Focused Analysis of Qualitative Interviews with MAXQDA: Step by Step. Peter Rawlins is an Associate Professor and Director of Academic Programmes for the Institute of Education, Massey University, New Zealand. Peter’s research interests are in assessment and mixed methods research. He teaches in undergraduate and postgraduate assessment, as well as a postgraduate mixed methods research course and a research project course. Linda A. Roussel served on the Improvement Science Research Network (ISRN) as Steering Committee member at the University of Texas Medical Center, San Antonio, engaging in research on operational failures (STAR-2) in healthcare systems. She is currently a member of a research team from the University of Alabama at Birmingham investigating methodological potentials and advantages of mixed methods research design for knowledge translation. Dr. Roussel has also developed, managed and coordinated Executive Nursing Leadership, Clinical Nurse Leader (CNL), and Doctor of Nursing Practice (DNP) academic programs. She was instrumental in developing the first DNP program in Alabama at the University of South Alabama and served as a key faculty in developing and growing the Clinical Nurse Leader (CNL) and Nurse Executive programs. Dr. Roussel is currently DNP faculty at the University of Texas Houston Cizik School of Nursing. Vanessa Scherman is a Professor in Psychology of Education at the University of South Africa and has been working in the field of mixed methods for a number of years. Drawing on her expertise are school effectiveness and psychosocial support, and drawing on data to make informed decisions. She has been project leader for a number of projects focusing on the design and development of interventions and instruments in both education and psychology. Previously, as part of her work on mixed methods school effectiveness studies, she worked extensively on the adaptation and implementation of monitoring frameworks, as well as exploring the influences of relationships within the school context on the achievement of learners. She has worked on a number of funded projects, including for the World Bank, United Nations Children’s Fund, National Research Foundation, as well as the Nelson Mandela Children’s Fund. She has also collaborated with national and provincial Departments of Education, and has served on national and international committees such as the UMALUSI Accreditation Committee, PSYSSA Research Methodology Division, as Chair of Governance and President of the Mixed Methods International Research Association (MMIRA). Judith Schoonenboom holds the chair for empirical pedagogy at the University of Vienna, Austria. She has extensive experience in designing and evaluating innovations in education. Judith specializes in mixed methods research in education, focusing on mixed methods design and the foundations of mixed methods research. Together with Burke Johnson, she wrote the highly successful article, “How to construct a mixed methods research design” (2017). Since 2014, Judith has held around 25 invited workshops on mixed methods research design. Peggy Shannon-Baker is an Associate Professor of Educational Research and an affiliate faculty member of Women’s, Gender, and Sexuality Studies at Georgia Southern University (USA). Their scholarship bridges two areas: systems of oppression such as racism and heteronormativity in education, and culturally relevant and sustainable research practices in mixed methods. This work has been published in the

Notes on the Editor, Section Leads, and Chapter Contributors

xxxi

Journal of Mixed Methods Research, International Journal of Qualitative Methods, and International Journal of Multiple Research Approaches, as well as edited volumes, such as The Routledge Handbook for Advancing Integration in Mixed Methods Research, and elsewhere. Dr. Shannon-Baker is also an Associate Editor of the Journal of Mixed Methods Research and has given workshops and guest lectures on mixed methods around the world. Mehdi Taghipoorreyneh is an Assistant Professor of Marketing at the University of Applied Science and Technology, Tehran. He uses mixed methods design as a methodological tool to understand the relationship between consumers’ cultural and identity backgrounds, and their response to new technology. He published a paper entitled “Using mixed methods research as a tool for developing indigenous cultural values instruments in Malaysia” in the Journal of Mixed Methods Research. In his previous work, he proposed mixed methods design as a new approach for assessing individuals’ attitudes toward social media advertisements. He works as Head of Marketing Research Department at Mines and Mining investment company, Sales Manager at Almas company, and as consultant for several energy companies. Currently, he is editorial board member of the Asian Journal of Business Research (AJBR), International Journal of Consumer Research (IJCR), and Borneo Journal of Social Sciences and Humanities (BJSSH). Talia Thompson is a licensed psychologist with Children’s Hospital Colorado and an Assistant Professor of Pediatrics at the University of Colorado School of Medicine. As a Qualitative and Mixed Methodologist, she supports child health researchers across the Anschutz Medical Campus with study design, data collection and analysis, and stakeholder-engaged program implementation. Dr. Thompson’s research focuses on enhancing quality of life and promoting psychological thriving for children with genetic differences and neurodevelopmental disorders. Susanne Vogl is a Professor of Sociology at the University of Stuttgart with a focus on research methods. Her research agenda is centered on improving existing methods in social sciences, and further developing techniques and methods. This line of research requires an in-depth understanding of data collection as a series of complex social interactions. Based on her aim to contribute to a more inclusive research practice, she develops integrative strategies for data collection and methods of analysis. Her work on interviewing special population groups, obtaining multiple perspectives, and conducting qualitative longitudinal research has led to philosophical and conceptual underpinnings of data integration in various forms, and contributes to advances in mixed methods analysis. Professor Vogl’s substantive fields of research include the sociology of deviance, children and young people, family, and life course. Daphne C. Watkins is a Professor of Social Work and a University Diversity and Social Transformation Professor at the University of Michigan. Professor Watkins developed the first Certificate Program in Mixed Methods Research, which was the motivation behind her first book Mixed Methods Research (2015, Oxford University Press). Professor Watkins also published a second book, Secondary Data in Mixed Methods Research in 2022 (Sage), after seeing exponential growth in resources using secondary quantitative and qualitative data, but not secondary data in mixed methods research. Professor Watkins is passionate about developing culturally appropriate strategies for conducting mixed methods and efficient data analysis tools, such as the Rigorous and Accelerated Data Reduction (RADaR) technique (2017), an individual and teambased approach for organizing, coding, and analyzing qualitative data. She currently directs the Vivian A. and James L. Curtis Center for Health Equity Research and Training at the University of Michigan. Xuejun Yin is a postdoctoral research fellow at the Chinese Academy of Medical Sciences and Peking Union Medical College. Her research has focused on improving health care and health systems, particularly in the cardiovascular health area through epidemiology, implementation science, and health policy research. She has experience with both qualitative and quantitative study designs, data analysis, and mixed methods for the implementation and evaluation of complex interventions. Dr. Yin completed her PhD in the George Institute for Global Health where she developed the evidence base for population-level salt reduction strategies for China through mixed methods. Lisa Zimmerman has had an interest in mixed methods for the last 15 years, and has used mixed methods in working with large-scale international studies of educational achievement, and in her work with studies in language and literacy curriculum implementation at school level, and testing and assessment systems. She teaches at the University of South Africa as an educational psychologist, working with undergraduate teacher education students, as well as teaching research methodology to postgraduate students.

Preface The Sage Handbook of Mixed Methods Research Design is a trailblazing edited book involving 80 chapter authors — the first to focus on community-sourced design topics and illustrative studies from around the world. Each of the 37 seminal chapters provides practical guidance for researchers tackling the complex processes involved in mixed methods research design. Many of the chapters point to essential new and emerging mixed methods design practices for researchers to know about and be able to perform. Importantly, the contributors to this Handbook are geographically diverse and represent both internationally renowned scholars, as well as those who are rapidly becoming established as innovators. As an essential resource for anyone interested in the contemporary, emerging, and evolving practice of mixed methods research and scholarship — a community-sourcing approach drew upon the expertise and experiences of our 12-member International Advisory Board to ensure the relevance of the design topics explored in this Handbook for our varied and dynamic audience needs. New and established researchers and evaluators will find the up-to-date literature as well as the historical and future-forward design discussions helpful for a field orientation. Instructors and learners enrolled in courses and workshops will find the in-depth descriptions and discussions of illustrative studies helpful for bridging theory with real world practice examples. Research teams will find the discussions and examples of innovative ways to collaborate on designs helpful for realizing desired outcomes. The book is organized into six sections led by an interdisciplinary group of 13 internationally renowned editorial section leads. To help researchers, the 35 chapters are organized in six sections with introductory and concluding chapters. • Handbook Introduction (Chapter 1) offers an overview of what you can expect, how the sections complement each other, and the unique perspectives afforded by a curated collection of chapters focused on mixed methods research design. • Section 1 (Chapters 2–6) relates the evolving dialogues from authors who have experienced and contributed to the many crossroads of mixed methods research design practices. • Section 2 (Chapters 7–12) details the craft attitude necessary for navigating specific components of the mixed methods research design process, such as emergence, sampling, data, ethics, visualization, and integration. • Section 3 (Chapters 13–21) expands what is currently known about how we might meaningfully use cultural contexts and intersections with other design types to inform mixed methods research practice innovations. • Section 4 (Chapters 22–26) leverages technology applications across design processes to advance new possibilities for innovative integrations. • Section 5 (Chapters 27–31) navigates research cultures in dynamic mixed methods research design processes to make explicit the ways our designs and team practices are informed by many cultural influences. • Section 6 (Chapters 32–36) explores emerging directions for design innovations by identifying and discussing how new things are coming together in ways we have not yet seen, or how existing things are coming together in new ways. • Handbook Conclusion (Chapter 37) explores the emerging and exciting future for mixed methods research design through discussions of four design topics and a concluding call for researchers to be bold and open in their thinking about design possibilities, be creative in their design practices, and be adaptive to their design contexts.

Acknowledgements The idea for this Handbook was the outcome of a conversation about the limited practical mixed methods research design guidance from a global perspective with Alysha Owens at Sage following the publication of my 2018 Innovation in Mixed Methods Research. I had the great fortune to have Colette Wilson and Umeeka Raichura as steadfast supporters throughout its development. Thank you very much for your guidance and patience and to everyone involved in this project at Sage. That such a Handbook was possible remains the consequence of the trailblazing efforts of Abbas Tashakkori and Charles Teddlie as the editors of the 2003 and 2010 editions of The Sage Handbook of Mixed Methods of Social & Behavioral Research and the many contributors whose chapters I have often referred to. I am indebted to the members of the global mixed methods community who have contributed, and continue to do so, to design practice advancements. My Editor role was made easier by the involvement of incredible chapter authors, section leads, and members of the International Advisory Board. Together with the Section Leads, we thank each of the 78 chapter authors for their insightful design work and willingness to engage in our review processes and respond productively to our comments. I thank each of the 13 section leads for their tireless efforts in supporting chapter authors and working with me to realize a truly unique and innovative vision for each of their sections. I also thank the 12 International Advisory Board members for their major role in supporting the inclusion of global and culturally diverse perspectives. To complement the International Advisory Board members, additional academic colleagues who appear in the list below offered their expertise and provided constructive comments during their review of chapters. My sincere apologies if I have inadvertently missed anyone. Sandra S. Abrams, St. John’s University Leena Åkerblad, University of Jyväskylä Louis Botha, University of the Witwatersrand Okan Bulut, University of Alberta Leia Cain, University of Tennessee, Knoxville Lina Eklund, Uppsala University José Luís Guedes dos Santos, Universidade Federal de Santa Catarina Jennifer Greene, University of Illinois Urbana-Champaign Will Mason, University of Sheffield The development of this Handbook happened while the COVID-19 pandemic had differential impacts around the world, from increased workloads to managing extreme and changeable conditions, including working from home and caregiving responsibilities. Throughout it all, we persisted with kindness and patience with one another as we each managed impossible situations. I am grateful to those who continue to influence my thinking about mixed methods research design— from those involved in this Handbook to the many learner participants who enroll in my mixed methods classes, workshops, and webinars, in addition to others with whom I interact in various places and ways. I thank the University of Alberta for their support in awarding me the McCalla Teaching Professorship (2020–2022) to focus on building mixed methods research capacity, and my department and faculty for their support to realize this project. Key among those who have helped me include three doctoral students: Alexandra Aquilina, whose early consultations provided the foundation for my editorial processes, Emily Mack, whose coordination of the peer review process was unparalleled, and Danae Strelau, whose keen eye helped finalize the Handbook materials for publication. To my many friends and colleagues locally and around the world, and my family – Joyce, Brian, Andrea, Lisa, Dennis, Anna, Thomas, Madison, and Jacob – thank you for your encouragement to pursue my own path. To my dad, Richard, whose influences can be seen in the way I live my life and who is lovingly remembered every day. Finally, to members of my Edmonton-based family – Damian, Avery, and Jasper – thank you for providing me the time and space to work with exceptional colleagues to create this Handbook. Thank you all. Cheryl N. Poth

This page intentionally left blank

1 Dilemmas and Opportunities for Mixed Methods Research Design: Handbook Introduction Cheryl N. Poth

The Sage Handbook of Mixed Methods Research Design (subsequently referred to as the Handbook) was developed to offer a global perspective of how researchers are leveraging the dilemmas and opportunities for mixed methods research design with the aim to inspire further innovations. The Handbook represents the culmination of sustained efforts over three years involving more than 80 mixed methods researchers from around the world to distil key learnings and guidance. Curating an inclusive collection was essential to capture the diversity of discussions and debates that abound in relation to just about everything about mixed methods research design—if you ask several researchers what it is, how we do it, where it begins and ends—I expect you will hear many different responses. Enriching our global mixed methods research community design conversation with learning about and from diverse perspectives contributes meaningfully to design practice innovations. My aim as Handbook Editor is to advocate for inclusion and innovation as I support and further mixed methods research design conversations that are useful for a global audience. The development of the Handbook was inspired by several questions including (but not limited to):

• What mixed methods research design perspectives would benefit others to learn from and advance the field? • What processes and outcomes ought to be involved in future-forward mixed methods research design practices? • What recent practice advances ought to be incorporated into the design of future-forward mixed methods research?

• What ought to be the scope of mixed methods research design?

By mixed methods research design, I refer to the emerging processes that initiate at a study’s

Design has been a frequently discussed and debated topic within the field of mixed methods research (e.g., Tashakkori et  al., 2021; Tashakkori & Teddlie, 2003, 2010) and I argue that the Handbook advances our design conversations in important and new ways. See the Handbook Conclusions (Chapter 37) for my revisit of these questions. I begin the Introduction to this Handbook with clarifying key terminology and perspectives that influenced its development.

KEY HANDBOOK TERMINOLOGY

2

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

conceptualization, that continue through to its conclusions and encompass the outcomes (e.g., proposals, publications) generated along the way. With this definition, I expand the conceptualization of mixed methods research design beyond the initial study formulation process and outcome. I use this definition to further explicate my view of the design of mixed methods research as paradoxically feasible and unwieldy: Feasible because the design of mixed methods research has been widely described in the literature as necessary to initiate prior to beginning research for the purpose of planning research procedures, to continue throughout the research process for the purpose of describing the research logic, as well as to depict the procedures employed after the research is completed for reflecting on threats to validity and research integrity as well as for comparison with other similarly designed studies (e.g., Creswell & Plano Clark, 2018; Tashakkori

et  al., 2021). Unwieldy because the design of mixed methods research is so ubiquitous in the literature, it is often difficult to pinpoint exactly what it entails and how these understandings have evolved over time. (Poth et al., 2022, p. 274)

The paradox is evident within the wide-ranging design topics and illustrative examples discussed throughout the Handbook chapters. To that end, this Handbook explores and addresses the question: What ought to be the scope of mixed methods research design?

KEY HANDBOOK PERSPECTIVES I offer an overview of the Handbook’s key numbers in Figure 1.1 as evidence of its commitment to a global audience and practice-orientation. The

Figure 1.1  An overview of the Handbook by key numbers Note: *Total represents unique contributors as many of the editorial Section Leads and International Advisory Board members were also chapter authors. Source: Author created.

DILEMMAS AND OPPORTUNITIES FOR MIXED METHODS RESEARCH DESIGN

development of this Handbook involved more than 80 contributors who played various functions as chapter authors, Section Leads and International Advisory Board members, all aimed at increasing the relevance of this Handbook for a global audience. From its initial conception, I imagined this Handbook as an opportunity to bring together global perspectives in a curated collection to help mixed methods researchers navigate the design complexities they are likely to encounter. It was my intention for the Handbook to reflect diverse perspectives and divergent viewpoints because, as Fàbregues, Escalante-Barrios, et  al. (2021) noted: “such engagement will lead to valuable insights that could lay the basis for further discussion needed to ensure the healthy development of the field” (p. 2). We asked subject matter experts to use illustrative examples to advance practical guidance in their chapters about different aspects of the complex processes involved in mixed methods research design. Together, this collection of 37 chapters plus Introductions and Conclusions for each of the six sections point to established as well as essential new and emerging mixed methods design practices that researchers should know about and be able to perform. To that end, this Handbook explores and addresses the question: What mixed methods research design perspectives would benefit others to learn from and advance the field?

A commitment to inclusion of global perspectives can be seen in the country of primary affiliation of Handbook contributors shown in Figure 1.2. The shading represents the 16 countries from which the chapter authors hail, and icons distinguish the location of the Section Leads, the Editor and International Advisory Board members. Note that the icons do not differentiate if more than one contributor is from the same location. Importantly, in addition to the geographical diversity of Handbook contributors, career stages are also represented; these include internationally renowned scholars, as well as those who are rapidly becoming established as innovators. Noteworthy is the less familiar map which I have used to represent the geographical locations of Handbook contributors. Figure 1.2 uses a 1954 Buckminster Fuller Airocean projection of the world map onto the surface of an icosahedron, which is a variant of the Dymaxion map (also known as Fuller map) to reduce the distortion of the relative size of areas in a flat drawing. The intention was for this projection to avoid having a “right way up”, which supports the efforts to reflect diverse and novel perspectives in this Handbook. After more than two decades as an instructor, researcher, reviewer and author, I have come to recognize three essential influences on my thinking about mixed methods research design. I share

Figure 1.2  Geographical locations of Handbook contributors on a 1954 Buckminster Fuller Airocean projection of the world map Source: Author created.

3

4

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

them here to make explicit my epistemological orientation to the Handbook. 1 Integration as the distinguishing feature of mixed methods research design. Without evidence of integration in a design, I do not consider it to be mixed methods research. In this way, I preserve important distinctions between multi­ method and mixed methods that others have also made explicit in descriptions of key characteristics (i.e., Creswell & Plano Clark, 2018), thoughtful arguments (i.e., Greene, 2015), and discussions of mixed analysis strategies (i.e., Hitchcock & Onwuegbuzie, 2020). I also recognize that my thinking continues to evolve with experience and commit to making such evolutions explicit in my writing. A key example is that while I have defined mixed methods research design in the current Handbook as necessitating the integration of qualitative and quantitative data, I am becoming more open to arguments highlighting the constraints introduced by categorizing data as either qualitative or quantitative. I attribute this shift to my experiences with data formats, such as Tweets and videos, that, in my view, can transcend this dichotomy. I might need to revisit my current conceptions of integration within mixed methods research design in the near future. I consider monitoring evolutions in my thinking as a key part of engaging as a lifelong mixed methods research learner. 2 Representing diverse perspectives is key to creating an inclusive global mixed methods research community. Without seeking others who think differently or work in unfamiliar research contexts and valuing their perspectives, we risk ill-informed assumptions about the applicability of design practices across diverse contexts. Through seeking and valuing diverse perspectives, I commit to an inclusive approach where I advocate for opportunities for authors to represent voices, experiences and perspectives that have not previously been represented in publications such as this Handbook. I acknowledge that there remains much work to be done within the field of mixed methods research and the growing body of literature to guide this work— for example, mixed methods research design intersections with Indigenous, community-based, and participatory research approaches (e.g., DeJonckheere et al., 2019; Mertens, 2023). I am eager to learn from others and assimilate those

learnings in future researcher, author, reviewer and editorial roles. 3 Accessible opportunities for respectful sharing and discussing of emerging ideas are central to advancing mixed methods research design practice. As mixed methods research professionals, we risk stagnation and isolation without opportunities to share and receive feedback on our ideas. Technology-mediated interactions, such as webinars and emails, have greatly expanded the frequency and accessibility of opportunities that at one time could only occur in-person at conferences and meetings. I commit to engaging with others respectfully in forming relationships while interacting and supporting others. I am mindful of the need for and an advocate of public scholarship to increase audience reach through (but not limited to) open access mixed methods research-focused public talks and publications. Now to explore the why, what and how of the Handbook.

WHY DOES THE HANDBOOK FOCUS ON DESIGN? The Handbook’s focus on mixed methods research design is timely to promote innovation in our thinking and actions. The Handbook is made possible by building upon the previous contributions of others discussing the topic of mixed methods research design, including the trailblazing work of Abbas Tashakkori and Charles Teddlie as the Editors of the 2003 and 2010 editions of The SAGE Handbook of Mixed Methods of Social & Behavioral Research, and the many chapters and articles that have since been published. In a virtual special issue focused on the design of mixed methods research, I and colleagues describe some of the remaining design challenges that researchers face alongside a discussion of seminal and contemporary design advancements (Poth et  al., 2022). Key among those challenges involves the lack of transparency in mixed methods research design descriptions that is highlighted as a key quality criterion (e.g., Fàbregues, Molina-Azorin, et al., 2021; Hirose & Creswell, 2022; O’Cathain, 2010; Onwuegbuzie & Poth, 2016). Noteworthy, Guetterman and colleagues (2022) define quality as a broader concept than validity and legitimation that refers to “how well a mixed methods study was conducted through scientifically accepted

DILEMMAS AND OPPORTUNITIES FOR MIXED METHODS RESEARCH DESIGN

design and procedures” (p. 2). To this end, we support the advocacy efforts calling for authors to report the essential design aspects, including “what they did, how they did, and why they did it” (Zhou & Wu, 2022, p. 275). I, along with others, consider mixed methods research designs as prevalent, recognizable and rapidly evolving, and researchers need to be prepared for the changing conditions in which they will conduct studies. Access to information about how researchers plan for and navigate changeable conditions would help guide researchers in preparing more authentic design descriptions in their proposals and reports. Mixed methods researchers need to be able to think and act “complexively” and creatively in research design. Over the last decade, the ever-increasing recognition and use of mixed research approaches to solve complex societal issues have created great interest in and opportunities for novel thinking about designs. I advocate for further innovation in mixed methods research design as we tackle societal issues within increasingly complex conditions. The key characteristic of complex conditions is its changeability, meaning that mixed methods research designs can no longer be assumed to be fixed and conducted in predetermined ways. Instead, researchers must pay close attention and adapt appropriately to the changing study conditions. The adaptive practice ideas advanced in my book, Innovations in Mixed Methods Research: A Practical Guide to Integrative Thinking with Complexity (Poth, 2018), have gained acceptance as researchers increasingly recognize the complexity inherent in their studies. I point to the wise guidance from Pat Bazeley (2018) that “this complexity, evident across many fields of action and inquiry, demands methods able to investigate a problem from multiple viewpoints, with flexibility to adapt to changing situations, yet able to produce credible results convincing to diverse audiences” (p. 4). To that end, this Handbook explores and addresses the question: What processes and outcomes ought to be involved in contemporary mixed methods research design?

WHO ARE THE AUDIENCES FOR THIS HANDBOOK? As an essential resource for anyone interested in the contemporary, emerging and evolving practice of mixed methods research and scholarship, this Handbook is written for those with various roles and experience in mixed methods research

5

design. These include (but are not limited to) graduate students, instructors and learners of mixed methods research courses or workshops, individual researchers or members of a mixed methods research team, research associations and communities, research managers and reviewers of mixed methods research. In particular: • New and established researchers and evaluators will find the up-to-date literature as well as the historical and future-forward design discussions helpful for a field orientation. • Instructors and learners enrolled in courses and workshops will find the in-depth descriptions and discussions of illustrative studies helpful for bridging theory with real-world practice examples. • Research teams will find the discussions and examples of innovative ways to collaborate on designs helpful for realizing desired outcomes. To that end, this Handbook explores and addresses the question: What recent practice advances ought to be incorporated into the design of future-forward mixed methods research?

WHAT UNIQUE INSIGHTS DOES THIS HANDBOOK OFFER? Individual chapters offer practical guidance grounded in illustrated examples from authors’ own experiences and existing literature related to a specific mixed methods research design topic. Each of the six Handbook sections has its own Introduction highlighting the contribution of each chapter, which I encourage you to read. Here in the Handbook Introduction, Chapter 1 and in Figure 1.3, I offer an overview of each section by briefly describing what you can expect and how the sections complement each other and the unique perspectives afforded by a curated collection of chapters. By studying the information below and the individual section introductions, readers can locate the chapters relevant to their needs. Readers may also discover additional chapters that pique their interest. Readers should not feel restricted to reading the chapters in order, but instead to read ahead and return as they are compelled. The Handbook Conclusions (Chapter 37) allude to ideas woven across the sections to describe four emerging and exciting future directions for mixed methods research design conversations and practices.

6

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 1.3  Overview of Handbook chapters and unique contributions of sections Source: Author created.

Section 1: Inspiring Diversity and Innovation in Mixed Methods Design With a focus on design diversity and innovation, Section 1’s chapters relate the evolving design dialogues from authors who have experienced and contributed to the many crossroads of mixed methods research design practices. Section coleads Sergi Fàbregues and José F. Molina-Azorin begin the section Introduction, “Evolving Tensions and Conversations in Mixed Methods Research Design Approaches”, with an orientation to the diversity and lack of consensus around the construct of mixed methods designs and suggest that the continuing coexistence of multiple perspectives serves the field well. Chapters 2–4 discuss design typologies and approaches, whereas Chapters 4–6 focus on the design of mixed methods for advocacy and knowledge. In the Section 1 Conclusion, “Future Tensions and Design Conversations in the Mixed Methods Field”, Molina-Azorin and Fàbregues point to the highly changeable mixed methods research design

conceptions that are likely to evolve alongside design practice expansions. We can infer from this first section that mixed methods research design developments are likely to help researchers solve specific problems and promote a better, more equitable society. In Chapter 2, “Revisiting Mixed Methods Research Designs Twenty Years Later”, John W. Creswell and Vicki L. Plano Clark explore the evolution of their prevalent design typologies and reflect on their changing perspectives of mixed methods research design. In Chapter 3, “Mixed Methods Design in Historical Perspective: Implications for Researchers”, Joseph A. Maxwell makes the case for interactive design approaches that are more inclusive of studies that meet the criteria for mixed methods research (even when the authors may not identify them as such) as benefitting the field. Chapter 4, “Mixed Methods Designs to Further Social, Economic and Environmental Justice” by Donna M. Mertens, details the potential of mixed methods designs informed by a transformative lens to realize research that is closely

DILEMMAS AND OPPORTUNITIES FOR MIXED METHODS RESEARCH DESIGN

aligned with notions of social, economic and environmental justice. In Chapter 5, “Developments in Mixed Methods Designs: What Have Been the Dominant Pathways and Where Might They Take Us in the Future?”, after revisiting four key historical influences on the evolution of designs, Katrin Niglas encourages flexible thinking, and mixed methods research design adaptations and adjustments as useful for addressing wicked programs and grand societal challenges. In Chapter 6, “The Role of Methodological Paradigms for Dialogic Knowledge Production: Using a Conceptual Map of Discourse Development to Inform MMR Research Design”, Dawn Freshwater and Jane Cahill describe a conceptual map of discourse development. The conceptual map is used to explain the epistemological process and relational activity through which the mixed methods research paradigm and knowledge is generated, emphasizing critical reflection for informing research design.

Section 2: The Craft of Mixed Methods Research Design To detail the craft of mixed methods research design, Section 2’s chapters present innovative approaches and practical guidance for navigating key design components. In Section 2 Introduction, “The Craft of Mixed Methods Research Design”, co-leads Sophia L. Johnson and Judith Schoonenboom aptly describe the craft attitude as needed for addressing the complexity inherent in designing and conducting mixed methods research. They frame the craft of mixed methods design as a product of the interplay among an iterative and adaptive decision-making process (techne), continuous reflection on the ethical implications of those decisions (phronesis), and general knowledge (episteme). In the Section 2 Conclusion, “The Craft of Mixed Methods Research Design”, Schoonenboom and Johnson highlight the key contribution of Chapters 7–12 to elucidate a craft approach to mixed methods design decisions, which are complex, multi-step, detailed, and iterative. The Section 2 chapters offer practical navigation guidance to researchers in crafting mixed methods research design. In Chapter 7, “Embracing Emergence in Mixed Methods Designs: Theoretical Foundations and Empirical Applications”, Manuela De Allegri and Julia Lohmann feature a discussion of definitions and theoretical foundations for emergent mixed methods designs, and guide its implementation through their illustrative healthcare design example in Malawi. In Chapter 8, “The Methods-Inference Map: Visualizing the Interactions Between Methods

7

and Inferences in Mixed Methods Research”, Judith Schoonenboom describes the methods-inference map using an illustrative secondary education example in New Zealand that offers practical visualization guidance for the interactions between methods and inferences in mixed methods research. In Chapter 9, “Towards Sampling Designs that are Transparent, Rigorous, Ethical and Equitable (TREE): Using a Tree Metaphor as a Sampling Meta-Framework in Mixed Methods Research”, Julie A. Corrigan and Anthony J. Onwuegbuzie demonstrate the TREE meta-framework for developing sampling designs using a multi-country higher education COVID-19 study. In Chapter 10, “Data Integration as a Form of Integrated Mixed Analysis in Mixed Methods Research Designs”, Susanne Vogl introduces a conceptualization of integration that considers the blurring boundaries between qualitative or quantitative data classifications. She uses focus groups with kids to illustrate design innovations addressing complex mixed methods research questions in action. In Chapter 11, “Ethical Issues and Practices for Mixed Methods Research in an Era of Big Data”, Roslyn Cameron and Heinz Herrmann weave ethical considerations within their discussion of a conceptual framework depicting the technology relationships in big data across two studies. These studies illustrate the Fourth Industrial Revolution era involving digital transformations, including the adoption of artificial intelligence and algorithms across systems and societies. In Chapter 12, “Building the Logic for an Integrated Methodology: Mixed Method Grounded Theory as an Example of Constructing a Methodology to Guide Design and Integration”, Elizabeth G. Creamer, Cherie D. Edwards, and Cassandra McCall demonstrate how to integrate mixed methods into another research methodology by identifying compatibilities and tensions between the philosophical assumptions and core procedures of the methodologies involved. Their fully integrated approach to methodology integration embeds multilevel mixing from its conception, as demonstrated by their illustrative mixed methods– grounded theory study involving Canadian mental health organizations.

Section 3: Expanding Mixed Methods Design Approaches To expand what is currently known about how we might meaningfully use cultural contexts and intersections with other design types to inform mixed methods research practice innovations, Chapters 13–21 illustrate design intersections and cultural considerations in real-world research

8

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

contexts. In the Section 3 Introduction, “Expanding Beyond Typology-Based Mixed Methods Designs”, section co-leads Peggy Shannon-Baker and Jessica T. DeCuir-Gunby describe their aim for Section 3 as expanding typology-based mixed methods design across the two themes: cultural considerations in mixed methods design and design combinations of mixed methods with other methodological approaches. Together, the nine chapters feature embedded illustrative examples to showcase how their innovative designs look in practice across global contexts and how the design contributes to practice advancements beyond their current study contexts. In the Section 3 Conclusion, “Moving Beyond Tradition: The Need for Expanded and Culturally Relevant Mixed Methods Design Typologies”, DeCuir-Gunby and ShannonBaker imagine a future for mixed methods research with calls for more culturally relevant design practices throughout mixed methods research designs. In so doing, they imagine adapting mixed methods designs to their cultural contexts, encouraging participatory approaches, and increasing researcher reflexivity. Chapters 13–16 describe cultural considerations in mixed methods design. In Chapter 13, “Exploring Interlocking Relationships of Race, Gender and Class with an IntersectionalityInformed Mixed Methods Research Design Framework”, Jenny Douglas describes the contribution of the theoretical frame of intersectionality within a mixed methods study of cigarette smoking among African-Caribbean teen girls to the development of relevant and appropriate public health research and promotion initiatives addressing inequalities in healthcare delivery. In Chapter 14, “Indigenous Cultural Values Instrument Development: Using Mixed Methods Research”, Mehdi Taghipoorreyneh elucidates how a sequential mixed methods research design integrated the cultural values of three major ethno-cultural groups (i.e., Malay, Chinese and Indian) within the multicultural country of Malaysia. He details the mixed methods procedures involving a three-round Delphi study and correspondence analysis of scale for generating the necessary reliability and validity evidence for the Indigenous Cultural Values instrument. In Chapter 15, “What Can Mixed Methods Partnerships Learn from Kaupapa M¯aori Research Principles?”, Peter Rawlins, Philippa Butler, Spencer Lilley, and Maggie Hartnett describe the principles underpinning Kaupapa M¯aori research, an Aotearoa New Zealand Indigenous research approach, in the context of mixed methods research. The He Awa Whiria (braided river) metaphor, which conceptualizes how Western and Indigenous research approaches can work together to create a whole that is more

than the sum of the parts, holds great potential for guiding indigenous-involved and culturally appropriate mixed methods design practices that are transferable beyond their study context. In Chapter 16, “Prioritizing Cultural Responsiveness in Mixed Methods Research and Team Science with Underrepresented Communities”, Tera R. Jordan and Maya Bartel relate a team-based mixed methods research design prioritizing cultural responsiveness with underrepresented communities. The authors advance the need for effective cross-disciplinary teamwork to tackle increasingly complex societal issues and inequality in research. Gleaned from a comprehensive review of relevant literature and experiential reflection, this chapter offers novel insights, practical strategies and innovative solutions for engaging underrepresented communities in mixed methods research. Chapters 17–21 detail design combinations of mixed methods with other methodological approaches. In Chapter 17, “Using Participatory Methods in Randomized Controlled Trials of Complex Interventions”, Jenevieve Mannell and Audrey Prost highlight the enormous value of participatory qualitative methods in randomized controlled trials (RCT) for advancing mixed methods evaluation design practices of complex interventions. The authors’ illustrative example elucidates the embedded mixed methods design of a cluster RCT to evaluate an intervention to prevent violence against women in Samoa that uses participatory methods to ensure appropriate randomization procedures, inform outcome measurement and engage stakeholders in interpreting the results using visual images. In Chapter 18, “Illustrating the Mixed Methods Phenomenological Approach (MMPR)”, Joanne Mayoh, Talia Thompson and Shanlee Davis advance the use of MMPR for complementarity purposes and demonstrate its potential for supporting equity, diversity and inclusion using an example to explore physical activity and quality of life in adolescents with Turner syndrome. In Chapter 19, “Intersection of Mixed Methods and Case Study Research (MM+CSR): Two Design Options in Educational Research”, Loraine D. Cook and Vimala Judy Kamalodeen use higher education examples from the Caribbean context to distinguish the key features and future potential for case study mixed methods (CS-MMR) and mixed methods case study (MM–CSR). In Chapter 20, “Harnessing Mixed Methods for Research Instrument Development and Legitimation”, Vanessa Scherman and Lisa Zimmerman describe the value of integration and advance practical tools for guiding mixed methods instrument development. In Chapter 21, “Mixed Methods-Grounded Theory: Best Practices for Design and Implementation”, Michelle C. Howell,

DILEMMAS AND OPPORTUNITIES FOR MIXED METHODS RESEARCH DESIGN

Wayne A. Babchuk and Timothy C. Guetterman advance a set of strategies, guidelines or best practices to help guide mixed methods-grounded theory research practice.

Section 4: Designing Innovative Integrations with Technology To design innovative integrations with technology and advance new possibilities, Section 4’s chapters leverage technology applications across design processes. In “Designing Innovative Integrations with Technology”, Section Lead Timothy C. Guetterman asks readers if they have “considered how technology can help to achieve integration in mixed methods research” (p. 311). He goes on to define integration and explain how the Section 4 description evolved to focus on technology after reading the initial chapter drafts. Together, the collection of Chapters 22–26 demonstrates how software, technology and technology-oriented research offer additional ways to mine and mix data. In the Section 4 Conclusion, “The Untapped Potential of Technology for Integration”, Guetterman outlines future directions to promote the use of technology in mixed methods research integrations. A unique contribution to the conversation involves his descriptions of advocacy roles for mixed methods research practitioners in the field, those who act as methodologists tasked with advancing practices, and software developers who are working on new ways to embed technology in mixed methods research design practices. In Chapter 22, “Using Software for Innovative Integration in Mixed Methods Research: Joint Displays, Insights, and Inferences with MAXQDA”, Udo Kuckartz and Stefan Rädiker explore the use of software in mixed analysis and identify interactivity in their illustrative examples as a key factor for software-based integration and joint display representation. In Chapter 23, “Grounded Text Mining Approach: An Integration Strategy of Grounded Theory and Textual Data Mining”, Mitsuyuki Inaba and Hisako Kakai provide practical guidance through the description and real-data illustration of four iterative stages of a grounded text-mining approach using focus group interview data obtained from Japanese graduate students and professors of nursing. In Chapter 24, “A ‘Mixed Methods Way of Thinking’ in Game-based Research Integrations”, Lisbeth M. Brevik uses game-based mixed methods research with teenage boys as co-researchers in Norway to demonstrate the value of integrating various voices and perspectives—and the differences

9

between them—when studying complex phenomena. In Chapter 25, “Integrating Secondary Data from Ethnically and Racially Minoritized Groups in Mixed Methods Research”, Daphne C. Watkins and Natasha C. Johnson examine the challenges and propose solutions for guiding the integration of secondary data from ethnically and racially marginalized groups into mixed methods research designs. In Chapter 26, “Beyond the Joint Display in Mixed Methods Convergent Designs: A CaseOriented Merged Analysis”, Carolina Bustamante offers practical guidance for selecting cases of exceptional teachers of Spanish participating in an online professional development program and using graphics to facilitate insights.

Section 5: Navigating Research Cultures in Mixed Methods Design To guide how researchers navigate research cultures in mixed methods research, Chapters 27–31 explicate how research culture influences designs across global research contexts. Authors were asked, to the extent possible, to make explicit the ways their designs are informed by cultural influences and how their designs and practices adapted to their unique research contexts. Section co-lead Elizabeth Creamer in “From Margin to Center: The Design Implications of Cultural Component with Mixed Methods”, highlights the elusive nature of the concept of culture and the difficulties faced by researchers in recognizing cultural influences on their designs. In the Section 5 Conclusion, “Future Directions for Research Cultures in Mixed Methods Designs”, section co-lead Elsa Lucia Escalante-Barrios recognizes this section’s significant and relevant contributions to design conversations advancing complex designs where features were determined by unique characteristics of cultural groups. She also emphasizes the need to make visible the ongoing methodological dialogues to consolidate new ways of implementing mixed methods. In Chapter 27, “Culturally Responsive Mixed Methods Evaluation Design”, Jori N. Hall and Ayesha S. Boyce discuss the essential role of researcher reflexivity in their illustrative example of a culturally responsive mixed methods evaluation of a US-based STEM project. A key contribution of this chapter involves recognizing the various influences of culture on and within the evaluation from the individuals (i.e., evaluators and stakeholders) involved to the geographical (i.e., US) and disciplinary (i.e., STEM) contexts for the evaluation. Importantly, Hall and Boyce call for considering

10

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

the applicability of their culturally responsive mixed methods stance with values-engaged, equity-focused, and anti-racist commitments and acknowledge culturally responsive stances as necessary and credible frameworks for mixed methods inquiry. In Chapter 28, “Integrating a Four-Step Japanese Cultural Narrative Framework, Ki-Shou-Ten-Ketsu, into a Mixed Methods Study”, Taichi Hatta explicates how a cultural narrative framework from Japan, ki-shou-ten-ketsu, informed the analysis of a qualitative dialogue in a mixed methods study of cancer treatment. In Chapter 29, “Leveraging Mixed Methods Community-based Participatory Research (MMCBPR) in Diverse Social and Cultural Contexts to Advance Health Equity”, P. Paul Chandanabhumma, Annika Agni and Melissa DeJonckheere illustrate the value of MMCBPR to meaningfully engage socially and culturally diverse US Indigenous communities and advance well-being and health equity. In Chapter 30, “Cultural Diversity in Intervention Designs: A Chinese Illustrative Example”, Hongling Chu, Xuejun Yi and Huieming Liu illustrate the impacts of cultural diversity on large-scale mixed methods intervention studies by describing a health-prevention process evaluation conducted in 120 villages across five provinces in culturally diverse rural China. In Chapter 31, “Examining the Influences of Spanish Research Culture in Systematic Observation with Mixed Methods”, M. Teresa Anguera, Eulàlia Arias-Pujol, Francisco Molinero and Luca Del Giacco explicate how observational methodology as mixed methods is influenced by the Spanish research cultural context where it takes place.

Section 6: Exploring Design Possibilities and Challenges for Mixed Methods Research for the Future To imagine future possibilities for mixed methods research, Chapters 32–36 explore emerging directions for design innovations by identifying and discussing how new things are coming together in ways we have not yet seen, or how existing things are coming together in new ways. In the Section 6 introduction, “Exploring Possibilities and Challenges for Mixed Methods Research for the Future”, section co-leads Peter Rawlins and Maggie Hartnett orient readers to three threads (i.e., evidence, people, and technology) and the interconnections among those threads that they asked authors to address. In the Section 6 Conclusions, “Where to Next in Exploring

Possibilities and Challenges for Mixed Methods Research for the Future?” Rawlins and Hartnett revisit the key threads to discuss the ideas advanced by each of the chapters and connect them with their own lived experiences. In Chapter 32, “Visualizing the Process: Using Visuals to Teach and Learn Mixed Methods Research”, Peggy Shannon-Baker demonstrates their use of visuals for building emerging researchers’ capacities. In Chapter 33, “Toward the Future Legitimacy of Mixed Methods Designs: Responsible Mixed Methods Research for Tackling Grand Challenges for the Betterment of Society”, José F. Molina-Azorin and Michael D. Fetters advocate the use of mixed methods designs for the betterment of society and guide researchers in achieving academic, practical and social impact using illustrative examples addressing timely grand challenges. In Chapter 34, “Realizing Methodological Potentials and Advantages of Mixed Methods Research Design for Knowledge Translation”, Nataliya V. Ivankova, Jami L. Anderson, Ivan I. Herbey, Linda A. Roussel and Daniel Kim enhance translational research by its intersection with mixed methods as an integrative thinking guide for addressing complex knowledge translation problems. In Chapter 35, “Opportunities and Challenges for a Transdisciplinary Mixed Methods Research Future”, Mandy Archibald relates the possibilities of transdisciplinary mixed methods research arguing its usefulness as a collaborative approach for addressing wicked social problems. In Chapter 36, “Mapping Design Trends and Evolving Directions Using This Handbook”, John Creswell, Cheryl N. Poth and Peter Rawlins convey mixed methods design trends represented by the Sage Handbook of Mixed Methods Research Design and use a roadmap metaphor to describe four evolving directions for mixed methods research design.

Handbook Conclusions Finally, I conclude the Handbook with a short chapter, “An Emerging and Exciting Future for Mixed Methods Research Design: Handbook Conclusions”. In discussing four design topics, I weave my own perspectives with ideas alluded to in the Handbook sections and individual chapters illustrating some evolving landscapes of design terminology, illuminating many complex influences on design practices, representing diverse design perspectives and assimilating practice evolutions in design education. I speculate about the challenges likely to be encountered for each topic

DILEMMAS AND OPPORTUNITIES FOR MIXED METHODS RESEARCH DESIGN

and suggest actions aimed at moving the field forward. I conclude by advocating that both creativity and openness are vital for inspiring the design of mixed methods research applicable for global contexts and the education of mixed methods researchers capable of design innovations for the yet unknown future.

11

opportunities for new perspectives and contributors to become involved in future discussions and publications, and for the global mixed methods research community to benefit and learn from the description of this Handbook’s development.

Key Events, Decisions and People WHAT INFLUENCED THIS HANDBOOK’S DEVELOPMENT? Central influences on the development of this Handbook involve key events, decisions and people as well as a commitment to diversity, equity and inclusion (DEI). While my description of these influences represents a departure from what is typical to include in a handbook’s introductory chapter, my inclusion reflects a commitment to transparency about this Handbook’s development. It is my hope that making explicit details, such as how chapter topics were decided and how contributors came to be involved, can provide an example for others to follow in future publications. I hope it ultimately opens doors and

As I reflected upon the key events, decisions and people influencing the development of this Handbook, I began to sketch what now appears as Figure 1.4. In representing the development as a path, it is important to note that the sequence of the key events is non-linear and both influencing to and influenced by the people involved in each key event. This Handbook resulted from a unique set of circumstances, involving myself as the Editor working closely with 11 dedicated Section Leads and a 12-member International Advisory Board who together have generated a product that otherwise would have been impossible. I use the metaphor of a symphony composer and conductor to describe my role as Handbook Editor. My editor role involved supporting the development of the sections and integrating them

Figure 1.4  Key events, decisions and people influencing the development of this Handbook Source: Author created.

12

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

into a cohesive handbook. Readers should note the essential role of the chapter authors to make possible the ‘performance’ of this Handbook. The launching event involved the editor developing a handbook proposal that was subsequently subject to peer review and feedback. The proposal required making the case for this Handbook’s focus on mixed methods research design and detailing potential chapter topics and contributors. In this work, I relied on two sources for inspiration: my previous interactions with global mixed methods research community members and my reading of novel mixed methods research publications. I tracked, over several previous years, ideas for design topics that emerged from conversations at conferences and during workshops as well as from reading literature. I also noted people I met and heard present at various in-person and virtual conferences and webinars who were introspective in their examination of design experiences, dilemmas, and insights. As an example, I consulted notes I made after seeking out to speak with Jenny Douglas in 2019 at the 3rd Mixed Methods Research Caribbean Conference held in Trinidad and Tobago about an intersectional-informed mixed methods study she had completed. Likewise, I spoke with Taichi Hatta at the 2019 Mixed Methods International Research Association Asia Regional Conference in Hamamatsu, Japan about his recent cultural narrative framework. Note that while these researchers joined this Handbook as subject matter experts and served as lead chapter authors, not all researchers who appeared in the initial proposal were then invited to contribute. I credit the community sourcing of contributors (see next section for details) that played an essential role in expanding my network of potential Handbook contributors. Having secured a publishing contract for this Handbook based on the initial proposal, I then recruited Section Leads and an International Advisory Board. As an interdisciplinary group of 11 established authors, Section Leads shaped each of the six Handbook sections and worked closely and tirelessly with me as Editor and with the chapter authors within their sections. The 12-member International Advisory Board representing a range of established and emerging scholars from varied disciplines and geographical locations helped identify design topics and potential chapter authors. They also played a vital role in supporting the development of chapters by providing constructive feedback as peer reviewers. A community-sourcing approach informed how we (i.e., the Editor, Section Leads and International Advisory Board members) worked

together to increase the real-world relevance of the design practices discussed in this Handbook. We did this by seeking input from diverse members of the global mixed methods research community about the design dilemmas they experienced and their perspective of areas of design practice in need of further development. We also sought ideas about who might be suitable for writing on these design topics, and developed systems to work as efficiently and effectively as possible with one another. Together, the input shaped the questions that Handbook sections sought to examine, the design topics covered by individual chapters, and the subject matter experts ultimately invited to contribute a chapter. The resulting Handbook reflects a mix of chapters that are authored by newly established or familiar authors, solely or in pairs and teams. These chapter contributors then worked tirelessly with the Editor, editorial Section Leads and International Advisory Board members for readers to benefit from their lived design experiences and expertise. Together, the Editor and Section Leads extended invitations to authors to submit chapter proposals on specific topics. We invited authors to negotiate the focus of the topic and describe the illustrative examples they would use to offer practical guidance. In recruiting authors and design topics for the Handbook, we were mindful to provide space for both new ideas and for disagreements with established ones. Once agreement about the foundational elements of the chapter was reached, a chapter template and writing guidelines were provided to the authors along with a submission deadline approximately six months henceforth. Throughout the initial chapter development, the Editor and Section Leads interacted regularly to discuss and support authors. Once chapters were deemed ready for peer review, a chapter author contract was extended and the Editor organized the in-depth review involving the Editor, Section Leads and two peers to provide constructive feedback to authors. Peer reviewers were usually International Advisory Board members. When required, we sought additional perspectives from specific content experts. An editorial letter was compiled to guide the chapter revisions. In some cases, more than one round of revisions was supported. As part of the finalizing process, the Editor embedded suggestions for cross-referencing among Handbook chapters. As Editor, I saw crossreferencing as important to help readers make connections within and across the Handbook chapters and sections. Once authors reviewed the cross-referencing suggestions and prepared files for submission, the chapter was deemed to be

DILEMMAS AND OPPORTUNITIES FOR MIXED METHODS RESEARCH DESIGN

finalized. Once all the chapters were submitted, production involving copy-editing, typesetting, proofing and marketing began!

13

psychotherapy interactions in Spain (Chapter 31). Again, there are many more to explore. I leave you here to embark on a mixed methods research design journey!

Actioning A Commitment to Diversity, Equity and Inclusion (DEI)

REFERENCES

Inclusion efforts of Handbook contributors with diverse perspectives, as well as Handbook chapters advancing equitable design practices and featuring illustrative examples reflective of the wide-ranging contexts in which mixed methods research design takes place, are a few of the ways we actioned DEI during its development and in this Handbook itself. I note the need for future efforts to further expand the diverse perspectives, equitable practices and illustrative examples included in this Handbook. I believe these efforts are vital to increasing the impact of mixed methods research. Chapters advancing equitable design practices can be seen in discussions of design approaches that empower stakeholders and promote ethical research practices appear across the Handbook leveraging— for example, an innovative TREE sampling framework (Chapter 9), Maori-involved mixed methods research in New Zealand (Chapter 15), student involvement as co-researchers (Chapter 24), integration of secondary data from ethnically and racially marginalized groups (Chapter 25), and mixed methods communitybased participatory research with diverse US Indigenous communities (Chapter 29). We have a lot to learn from experts advancing equitable design practices and impactful experiential accounts, and the Handbook aims to provide this access. Chapters featuring illustrative examples reflective of the wide-ranging contexts in which mixed methods research design takes place can be seen in the more than 50 unique study contexts providing practical guidance here. It is noteworthy that more than half of the illustrative examples featured in this Handbook take place in contexts beyond North America. This introduces readers to mixed methods research contexts that may be less familiar and gives access to experiential accounts of real-world design processes to help them navigate their own cultural research contexts—for example, obstetric services in Malawi (Chapter 7), cigarette smoking among African-Caribbean teen girls (Chapter 13), violence prevention against women in Somalia (Chapter 17), challenges experienced by nursing graduate students and professors in Japan (Chapter 23) and communications during

Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. DeJonckheere, R., S. T. Lindquist-Grantz, K. Haddad & L. M. Vaughn (2019). Intersection of mixed methods and community-based participatory research: A methodological review, Journal of Mixed Methods Research, 13(4): 481–502. https:// doi.org/10.1177/1558689818778469 Fàbregues, S., Escalante-Barrios, E. L., Molina-Azorin, J. F., Hong, Q. N., & Verd, J. M. (2021). Taking a critical stance towards mixed methods research: A cross-disciplinary qualitative secondary analysis of researchers’ views. PLoS ONE, 16(7), e0252014. https://doi.org/10.1371/journal.pone.0252014 Fàbregues, S., Molina-Azorin, J. F., & Fetters, M. D. (2021). Editorial: Virtual special issue on “quality in mixed methods research”. Journal of Mixed Methods Research, 15(2), 146–151. https://doi. org/10.1177/15586898211001974 Fetters, M., Wu, J. P., & Chandanabhumma, P. P. (2021). Words matter: Calling on the community of research to recognize, react to, and remove racializing research rhetoric. Journal of Mixed Methods Research, 15(1), 6–7. https://doi. org/10.1177/1558689820977233 Greene, J. (2015). Preserving distinctions within the multimethod and mixed methods research merger. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 606–615). Oxford. Guetterman, T. C., Molina-Azorin, J. F., & Fàbregues, S. (2022). The need to rigorously develop common quality guidelines for reporting mixed methods research. Journal of Mixed Methods Research. https://doi.org/10.1177/15586898221143561 Hirose, M., & Creswell, J. W. (2022). Applying core quality criteria of mixed methods research to an empirical study. Journal of Mixed Methods Research. https://doi.org/10.1177/15586898221086346 Hitchcock, J. H., & Onwuegbuzie, A. (2020). Developing mixed methods crossover analysis approaches. Journal of Mixed Methods Research, 14, 63–83. https://doi.org/10.1177/1558689819841782 Mertens, D. M. (2023). Mixed methods research. Bloomsbury Academic.

14

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

O’Cathain A. (2010). Assessing the quality of mixed methods research: Toward a comprehensive framework. In Tashakkori A., Teddlie C. (Eds.), The Sage handbook of mixed methods in social and behavioral research (2nd ed., pp. 531–555). Sage. https://doi.org/10.4135/9781506335193.n21 Onwuegbuzie, A., & Poth, C. (2016). Editors’ afterword: Toward evidence-based guidelines for reviewing mixed methods research manuscripts submitted to journals. International Journal of Qualitative Methods, 15, 1–13. https://doi.org/ 10.1177/1609406916628986 Poth, C. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Poth, C., Molina-Azorin, J. F., & Fetters, M. D. (2022). Virtual special issue on “Design of mixed methods research: Past advancements, present conversations,

and future possibilities. Journal of Mixed Methods Research, 16(3), 274–280. https://doi.org/10.1177/ 15586898221110375 Tashakkori, A. M., Johnson, R. B., & Teddlie, C. B. (2021). Foundations of mixed methods research (2nd ed.). Sage. Tashakkori A., & Teddlie, C. (2003; Eds.). The Sage handbook of mixed methods in social and behavioral research. Sage. Tashakkori A., & Teddlie, C. (2010; Eds.). The Sage handbook of mixed methods in social and behavioral research (2nd ed). Sage. Zhou, Y., & Wu, M. L. (2022). Reported methodological challenges in empirical mixed methods articles: A review on JMMR and IJMRA. Journal of Mixed Methods Research, 16(1), 47–63. https://doi. org/10.1177/1558689820980212

SECTION 1

Inspiring Diversity and Innovation in Mixed Methods Design

This page intentionally left blank

Evolving Tensions and Conversations in Mixed Methods Research Design Approaches: Section 1 Introduction S e r g i F à b r e g u e s a n d J o s é F. M o l i n a - A z o r i n INTRODUCTION The purpose of Section 1 is to reflect on the history and evolution of mixed methods designs and consider the diversity of past and current design conceptualizations. The authors were selected on the basis of their substantial contributions to the mixed methods movement over its 40-year existence. They represent a heterogeneous group reflecting the diversity of disciplines, authors’ geographical locations, philosophical stances, and theoretical perspectives that characterize the field. We begin this Introduction with a brief discussion of how mixed methods designs have been conceptualized in the literature, with a particular emphasis on the lack of consensus in the ongoing debate. We then outline the organization of the section, including a brief summary of the key ideas discussed in each chapter.

CENTRALITY OF MIXED METHODS DESIGNS Since the early 1980s, a diverse yet cohesive group of scholars have identified the procedural

and theoretical components of mixed methods research as a distinctive methodological approach (Tashakkori et  al., 2021). As argued throughout the chapters in this section, mixed methods designs have been one of the most frequently debated and controversial topics in mixed methods research. According to Creswell and Plano Clark (2018)—two of the authors contributing to this section—mixed methods designs are frameworks that help organize the numerous methodological decisions that researchers must make throughout all stages of a mixed methods study, including not only the initial stage of study planning, but also the subsequent stages of data collection, analysis, and quality assessment (Poth et  al., 2022). Thus, designs in mixed methods research play a crucial role in articulating the whole range of procedures that are unique to this methodology, including the integration of quantitative and qualitative data and the generation of meta-inferences from the integrated findings (Fetters, 2020). Mixed methods designs can also have a pedagogical function when used by researchers who are less familiar with this approach, helping them to properly apply the methodology and to avoid the significant methodological flaws often found in current mixed methods empirical practice.

18

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

DIVERSITY AND LACK OF CONSENSUS AROUND MIXED METHODS DESIGNS Discussions about what constitutes a mixed methods design reflect the intrinsic heterogeneity of the field. As described in several chapters in this section, two main conceptualizations of designs in mixed methods research have been suggested in the literature: typology-based and interactive approaches. Typology-based approaches—proposed by John Creswell and Vicki Plano Clark, among other authors—have been predominant in the field since the early 1990s owing to their accessibility on account of the way that they classify and characterize the range of mixed methods designs available to researchers (Creswell & Plano Clark, 2018). However, the typology-based approach has also been criticized for being unduly prescriptive and insensitive to the contingent nature of mixed methods practice (Guest, 2013; Maxwell, 2016). During the 2000s, in response to these criticisms, Joseph Maxwell (another contributor to this section) and other authors proposed system-based, interactive approaches as a more flexible alternative focused on the processual interconnectivity of the components of a research study. These two different approaches to mixed methods designs have peacefully coexisted, illustrating how fostering often divergent viewpoints has allowed the field to evolve in a healthy manner. For a further discussion of the expansion of mixed methods research designs, see also the Introduction to Section 3 of this volume.

ORGANIZATION OF THE SECTION To further reflect on the foundational aspects of mixed methods designs and to describe the

significance of diversity in the historical evolution and current debates on designs in this type of research, several contributions to this section were solicited from seven authors renowned for their leadership in the mixed methods field, as well as for their philosophical, theoretical, and methodological differences. Table S1.1 summarizes the chapters for the authors who agreed to participate in the section. In the first edition of the Handbook of Mixed Methods in Social & Behavioral Research, John Creswell and Vicki Plano Clark, along with other co-authors (Creswell et  al., 2003) introduced a typology of mixed methods designs that has been widely adopted in the field due to its simplicity and specificity. In the first section of this handbook, these authors explore the evolution of their 2003 typology and reflect on the evolution of mixed methods designs since the release of the original version of this typology. They describe how their intention to distinguish mixed methods research designs from quantitative and qualitative approaches represented a journey “into the unknown” that helped to establish the foundational principles of the methodology. The authors also describe how their perspectives on designs have evolved over the past two decades, including a diminished emphasis on the concept of priority, the recognition that integration occurs through a wide variety of strategies at multiple stages, the emphasis placed on theory and meta-inferences, and a new classification of complex designs based on combining core designs with other methodological and/or theoretical frameworks. Joseph Maxwell has concentrated most of his work on proposing an alternative approach to study design based on a broader understanding of that concept. In several of his works (Maxwell, 2016; Maxwell et  al., 2015), he has argued that mixed methods designs should be articulated through a network of relationships between all the components of a study, as opposed to being

Table S1.1  Summary of Section 1 chapters: Inspiring Diversity and Innovation in Mixed Methods Design Chapter authors (country affiliation)

Chapter title

John W. Creswell and Vicki L. Plano Clark (USA) Joseph A Maxwell (USA)

Revisiting Mixed Methods Research Designs Twenty Years Later Mixed Methods Design in Historical Perspective: Implications for Researchers Mixed Methods Designs to Further Social, Economic, and Environmental Justice Developments in Mixed Methods Designs: What Have Been the Dominant Pathways and Where Might They Take Us in the Future? The Role of Methodological Paradigms for Dialogic Knowledge Production: Using a Conceptual Map of Discourse Development to Inform Mixed Methods Research Design

Donna M. Mertens (USA) Katrin Niglas (Estonia) Dawn Freshwater (Australia) and Jane Cahill (UK)

SECTION 1 INTRODUCTION

a collection or categorization of types, as emphasized in the typological approach. Adopting a different viewpoint from the one expounded in the first chapter of this section, Maxwell provides an overview of the history of mixed methods from BC to the present, demonstrating that this methodological approach had been used in studies well before the institutionalization of the movement in the 1980s. Through this historical overview, Maxwell shows that these studies were carried out without employing a typology of designs or referring to mixed methods-specific paradigms, and that the current typologies could not have captured certain forms of integration used in these studies. According to Maxwell, design approaches not constrained by typologies would allow the field to be more inclusive of studies that meet the criteria for mixed methods research, even when the authors may not identify them as such. Donna Mertens has been one of the most prominent proponents of the transformative approach in the field of mixed methods. In prior works, Mertens characterized the transformative paradigm as “a meta-physical umbrella that brings together many philosophical strands where social justice operates as a first principle” (Mertens & Cram, 2015, p. 92). In her chapter, Mertens discusses the topic of social justice, adopting an approach to conceptualizing designs. In particular, she describes how incorporating a transformative lens into mixed methods designs might result in research that is more closely aligned with notions of social, economic, and environmental justice. In her doctoral dissertation, Katrin Niglas (2004) examined the application of mixed methods designs in the field of education. This work, which was carried out at Tallinn University, was one of the first systematic assessments of the prevalence and implementation of mixed methods designs in a particular discipline, and it inspired the development of similar analyses in other fields. Adopting a broader viewpoint than that found in the previous chapters, in her chapter, Niglas revisits four key historical influences that have affected the evolution of mixed methods designs: design typologies, educational texts on mixed methods research, prevalence studies, and alternative conceptualizations of mixed methods designs. In line with Joseph Maxwell’s previous call for significant open-mindedness in the field, Niglas argues that, in order to ensure the successful development of the field, mixed methods researchers must be attentive to other ways of practising the methodology, especially across different cultures and disciplines, as well as to the changes brought about by innovative research practices. In a postmodern, reflexive critique of the language used to characterize mixed methods research

19

and its underlying assumptions, Dawn Freshwater (2007) advised against accepting mixed methods as the dominant discourse. Following up on this idea, in their chapter, Freshwater and Cahill propose a conceptual map of discourse development to explain the epistemological process and relational activity through which the mixed methods research paradigm is generated. This map may be especially useful for questioning existing assumptions in the mixed methods field and educating academics and stakeholders to critically examine the processes of knowledge generation. In addition, by enabling self-criticism, the conceptual map may help promote mixed methods research that embodies the ideals of social justice and equity. In contrast to the previous chapters, Freshwater and Cahill take a more philosophically oriented and critical approach to conceptualizing mixed methods designs.

CONCLUDING THOUGHTS Due to differences in research cultures and philosophical orientations, the authors in this section offer a wide and valuable range of perspectives on the topic of mixed methods designs, revealing a number of substantial disagreements on fundamental issues. On the subject of history, for instance, Creswell and Plano Clark date the first significant work on the subject of designs to the late 1980s in accordance with their emphasis on the concept of typologies and procedures unique to mixed methods, whereas Maxwell presents a historical account beginning in antiquity that is consistent with his broader view of designs. These divergences on foundational perspectives demonstrate a lack of homogeneity in mixed methods practice and thinking, which, rather than being a cause for concern, is indicative of the open-minded nature of the field. With these different views of mixed methods designs, we would like to highlight the openness of the field, as well as the ways in which the coexistence of multiple perspectives have contributed to the development of a more self-critical and mature mixed methods community.

REFERENCES Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Creswell, J. W., Plano-Clark, V., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed methods

20

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209–238). Sage. Fetters, M. D. (2020). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Sage. Freshwater, D. (2007). Reading mixed methods research: Contexts for criticism. Journal of Mixed Methods Research, 1(2), 134–146. https:// doi.org/10.1177/1558689806298578 Guest, G. (2013). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7(2), 141–151. https:// doi.org/10.1177/1558689812461179 Maxwell, J. A. (2016). Expanding the history and range of mixed methods research. Journal of Mixed Methods Research, 10(1), 12–27. https:// doi.org/10.1177/1558689815571132 Maxwell, J. A., Chmiel, M., & Rogers, S. E. (2015). Designing integration in multimethod and mixed methods research. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of multimethod and

mixed methods research inquiry (pp. 223–239). Oxford University Press. Mertens, D. M., & Cram, F. (2015). Transformative and indigenous frameworks for multimethod and mixed methods research. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 91–109). Oxford University Press. Niglas, K. (2004). The combined use of qualitative and quantitative methods in educational research. Tallinn Pedagogical University. Poth, C., Molina-Azorin, J. F., & Fetters, M. D. (2022). Virtual special issue on “Design of mixed methods research: Past advancements, present conversations, and future possibilities”. Journal of Mixed Methods Research, 16(3), 274–280. https://doi. org/10.1177/15586898221110375 Tashakkori, A., Johnson, R. B., & Teddlie, C. (2021). Foundations of mixed methods research. Integrating quantitative and qualitative approaches in the social and behavioral sciences (2nd ed.). Sage.

2 Revisiting Mixed Methods Research Designs Twenty Years Later J o h n W. C r e s w e l l a n d V i c k i L . P l a n o C l a r k

In 2003, we authored a chapter for Tashakkori and Teddlie’s Handbook of Mixed Methods in Social and Behavioral Research on “advanced mixed methods research designs” (Creswell et al., 2003). It was our first endeavour to pull together our thoughts on mixed methods designs, and it was a good forum for presenting our ideas. The Handbook represented the first major book on mixed methods research that surveyed the field and included an overview of mixed methods from many perspectives within its 768 pages. We believed that the Handbook stood as a watershed event in the field of mixed methods, and we intended to provide a state-of-the-art discussion about mixed methods designs. Our chapter set in motion for us much of our future writing. We began to specialize in mixed methods designs and have, overall, stayed with that topic as one of our primary areas of interest. The purpose of this chapter is to offer reflective commentary about mixed methods designs as we described in 2003 and as they have continued to evolve to the present day. We revisit our 2003 writing on mixed methods designs, situate our writing within the context of the development of mixed methods research, and discuss changes that have occurred in mixed methods designs over the last 20 years. We conclude with a few thoughts about challenges to a design approach in mixed methods and future directions for design-thinking.

Throughout this chapter, we share our ideas, acknowledging that others may hold different opinions about what has transpired. This reflection adds a notable perspective to the field by illustrating how thinking about mixed methods designs has evolved (and continues to evolve). Furthermore, it provides an overview of state-of-the-art thinking about designs with current references. Finally, it can assist current researchers in their design, conduct, and evaluation of mixed methods designs.

THE SURROUNDING CONTEXT OF MIXED METHOD DESIGNS OVER TIME As we now look back, the 2003 Handbook summarized the young, emerging field of mixed methods research and portended many events to come during the subsequent 20 years. Up to that point, only a few books and journal articles existed in the field. In 1989, Greene and colleagues provided a review of mixed methods evaluation studies, and in 1988 Bryman wrote about quantitative and qualitative research in his first book in the field. Miles and Huberman (1994) authored a book about qualitative data analysis with discussions that linked quantitative and qualitative data. A few scholars had identified types of mixed methods

22

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

designs, such as Creswell’s (1994) book on research design, articles by Morse (1991) and Morgan (1998), and Patton’s (1990) work in evaluation. These efforts were followed by Greene and Caracelli’s (1997) volume on mixed methods in evaluation and Tashakkori and Teddlie’s (1998) book on mixed methods as a methodology. It was then another five years before Tashakkori and Teddlie (2003) edited the first comprehensive handbook on mixed methods. Our 2003 chapter represented for us wandering into the unknown as we attempted to discuss foundational designs that we felt were useful in the mixed methods field. At that time, we were well aware of the use of specific designs in quantitative research, such as true experiments, quasiexperiments, surveys, and single subject designs, and in qualitative research, such as ethnography, grounded theory, phenomenology, and case studies. Campbell and Stanley (1963) had identified types of quantitative experiments and Creswell (1998) identified types of qualitative designs. We felt it was only a matter of time until the field of mixed methods delineated its own specific designs. After 2003, many developments in mixed methods unfolded to expand the field from its embryonic state, to adolescence, and now a sense of maturity (Molina-Azorin & Fetters, 2022). The Journal of Mixed Methods Research (JMMR) began in 2007 and rapidly developed into a leading social science journal. By 2013, the Mixed Methods International Research Association was founded to draw together scholars interested in or conducting mixed methods research from around the world. First in 2011 and then reissued in 2018, the National Institutes of Health (NIH) Office of Behavioral and Social Sciences Research (OBSSR) formed a committee that developed best practice recommendations for developing and reviewing mixed methods grant applications in the health sciences in the United States. In 2015, NIH also funded a major training program, located at Johns Hopkins, Harvard, and Michigan, to train emerging scholars in the health sciences in mixed methods research (https://publichealth. jhu.edu). In 2018, the American Psychological Association (APA) commissioned a task force to create “standards” for publishing mixed methods research. The work of this commission resulted in an article advancing the “standards” (Levitt et  al., 2018) and in guidelines included within the Publication Manual of the American Psychological Association (APA, 2020). Since APA’s Publication Manual is used widely around the world, this publication greatly expanded the reach of mixed methods. Along with these developments have come thousands of empirical mixed methods articles

in academic journals, increased funding of mixed methods projects by the United States’ federal government, and the delineation of two categories of journal publications—empirical articles that use mixed methods and methodological articles that advance the knowledge of the mixed methods field (Fetters & Molina-Azorin, 2019b). The books on mixed methods number over 31, as noted by Onwuegbuzie in 2012, and have expanded an estimated tenfold since then, according to a historical review of mixed methods books by Molina-Azorin and Fetters (2022). In addition, SAGE started a popular Mixed Methods Research Series edited by Plano Clark and Ivankova (e.g., including Plano Clark & Ivankova, 2016). Mixed methods research has expanded around the world with regional mixed methods conferences developing, and workshops and presentations in countries stretching from Asia to Africa. In short, the field has expanded considerably in its dissemination of information as well as its content since 2003. With all this expansion and development, it is timely to reflect on how understanding of mixed methods designs has developed and evolved during this time period.

SYNOPSIS OF OUR 2003 CHAPTER: ADVANCED MIXED METHODS RESEARCH DESIGNS To start our reflections, we first provide an overview of the major sections of our chapter on mixed methods designs (Creswell et al., 2003) and commentary about its content.

Major Characteristics of a Mixed Methods Study We began our 2003 chapter with a review of the major characteristics of an article by Hossler and Vesper (1993) reporting a study that examined parental savings for children attending higher education. This article was not identified as a type of mixed methods design back in 1993, but in the chapter we suggested that it could be called a “concurrent triangulation method design” (p. 210), which is a name we would not use today. We pointed out that the authors collected both quantitative (survey) and qualitative (interview) data. The title to the article also suggested both exploring and identifying factors—both strong qualitative and quantitative elements. In what we would now call a “rationale” for using mixed methods, the authors

Revisiting Mixed Methods Research Designs Twenty Years Later

indicated that the interviews would be compared with and provide a deeper understanding of the survey results. We also noted that the authors analyzed the two datasets separately, but discussed a comparison of the two databases which we called “integration” in 2003, the term we still use today. We ended this opening section by recommending that Hossler and Vesper label their type of design, identify the characteristics of the design, give it a name, and create a visual representation of their design. Most importantly, we called for authors of mixed methods studies, in general, to consider mixed methods as having “distinct” designs. At this early juncture in our thinking about designs, we foreshadowed elements that would be expanded in future years. Also, we recognized that readers were not familiar with mixed methods studies because of their paucity in the literature at this time, and we aimed to show what a mixed methods study looked like by starting with an example.

Key Elements of a Mixed Method Design In our next section of the 2003 chapter, we bridged the idea of mixed methods research and designs. After noting that designs were available in both quantitative and qualitative research, we defined “design” as research procedures, including data collection, data analysis, and interpretation, a workable definition that we still use today. We advanced the key elements in these procedures as implementation of data collection (concurrent or sequential), the priority in a mixed method report (equal, qualitative or quantitative), the place where mixing occurred in a study (data collection, data analysis and/or interpretation), and the use of a theoretical perspective (explicit or implicit). As we reflect on this passage, we still adhere today to our definition of design, but have changed rather dramatically our thinking about the key elements of a mixed methods design, especially those related to priority and theoretical perspective, and even timing to some extent. We see today that mixing or integrating does occur in different stages of the research, as well as the need to be specific about the use of theory in a design.

Literature Review Addressing Key Elements We next reviewed the literature about mixed methods that would inform our discussion of designs.

23

The authors writing at this time (1989–1999) helped us to focus on key elements to discuss about designs. For example, Steckler et al. (1992) provided some basic visual models of designs. Morse (1991) identified the use of lower case or upper case letters for QUAN or QUAL to identify the emphasis (or priority) given to each form of research in the mixed methods design, and suggested that designs fell into those that were simultaneous and sequential in timing. Morgan (1998) elaborated on this idea by identifying designs using different combinations of sequence, and the priority of quantitative and qualitative research. Tashakkori and Teddlie (1998) discussed the integration of the data in different stages of the research, such as in the research questions, the data collection, the data analysis and the interpretation. Greene and Caracelli (1997) added the importance of theory and the transformationalvalue of the dialog across ideological differences to encourage advocacy and empowerment. We included a table of the various typologies of designs available at that time, and noted eight different discussions of types of designs, their discipline origin, and the names for the designs. Today, most of these design elements are still applied in practice, although the use of notation has dwindled and the use of priority (dominance or lack of dominance of one type of data over the other) continues to be debated.

Six Mixed Methods Designs Given the many typologies of mixed methods designs and the major elements being pointed out by authors, we then chose to focus on six designs that reflected the criteria of implementation (timing), priority (emphasis), integration and theory use. With six designs, even back in 2003, we wanted a parsimonious set of designs that give researchers “the flexibility to choose and innovate within the types to fit a particular research situation” (Creswell et al., 2003, p. 223). The six designs were: sequential explanatory, sequential exploratory, sequential transformative, concurrent triangulation, concurrent nested and concurrent transformative. We discussed the major features of each of these designs. As we look back now, we introduced some vague thinking into these designs by focusing on priority as an important feature (equal, unequal), borrowing a term from qualitative research (triangulation), and relying heavily on more abstract, philosophical ideas by Greene and Caracelli (1997) of action-oriented designs to advocate or create empowerment.

24

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Design Issues in Need of Addressing We next discussed specific issues that these designs raised. One issue was the question of the paradigm or philosophical standpoint reflected in the designs with an emphasis on pragmatism (Tashakkori & Teddlie, 1998), and also consideration of how different designs could be used with different paradigms. The issue of specific data analysis procedures with the designs also caught our attention, and whether a computer program might be useful to aid in the analysis. We also suggested that the visualizations of designs were too basic, and that diagrams, for example, needed to be included and expanded to consider phases in the design. Returning to the Hossler and Vesper (1993) study, we suggested how their project might have been described using a visual model; a specific name for their design; the elements of data priority, sequence, and integration; and a transformative perspective. What can we say now about perspectives that have emerged since 2003? We have seen new names for the types of mixed methods designs, a greater focus on “intent” of the designs rather than timing and priority, more detail in the visualizations of designs, and simplified basic characteristics that underlay all designs. These are the changes we highlight in our next section.

CHANGING MIXED METHODS DESIGNS (2003–2023) Our attempt to catalogue the changes in mixed methods designs over the last 20 years is neither a definitive classification nor a complete rendering of possibilities. It does stem from our interest in the “methods” of mixed methods. Other authors address important topics in the field such as philosophical foundations, the connection of the stages of a research process, even specific issues such as validity and ethics. However, we believe that a central, even defining, feature of mixed methods resides in the topic of the designs, with many features emanating from it, such as the title of the study, the research questions asked, and the compositional structure of writing for publication. This may well be our myopic view of the field of mixed methods, but one that we have grown to appreciate. We organize our discussion of changes within four domains: definitions of mixed methods research, the names and elements of designs, diagrams of designs and integration within the designs.

Core Characteristics and Definitions of Mixed Methods Research Over the years, new concepts have flowed into the definition of mixed methods research. We have attempted to simplify the definition by focusing on the core characteristics. We begin where we started in our 2003 chapter with a discussion about definition. As we look back to this earlier chapter, today we are still using our basic definition of designs in mixed methods to include a focus on the data collection, data analysis and interpretation stages of a mixed methods study. Over the years we have called these stages the procedures for conducting a study. However, these procedures do reside within the larger picture of a definition of mixed methods research. What are the central features of mixed methods research? These features are embedded within definitions of mixed methods research, that, as Johnson et  al. noted in 2007, assumed different perspectives depending on the views of mixed methods scholars. We can map the evolution of definitions of mixed methods research beginning with an early definition by Greene et  al. (1989) who wrote: In this study, we defined mixed-methods designs as those that include at least one quantitative method (designed to collect numbers) and one qualitative method (designed to collect words), where neither type of method is inherently linked to any particular inquiry paradigm. (p. 256)

Looking back at this definition, we can see that it focused on the methods, as well as the paradigm or philosophy linked to the methods. This definition reflected at that time the on-going debate about paradigms, and the search for philosophical stances that undergirded mixed methods research. We next jump forward to our 2003 definition of mixed methods and compare it to our 2018 definition from our book about designing and conducting mixed methods research (Creswell & Plano Clark, 2018). We wrote in 2003: A mixed methods study involves the collection or analysis of both quantitative and/or qualitative data in a single study in which the data are collected concurrently or sequentially, are given a priority, and involve the integration of the data at one or more stages in the process of research. (Creswell et al., 2003, p. 212)

Here we see the introduction of the concept of integration as a central feature of mixed methods, as well as a focus on the timing and priority of the

Revisiting Mixed Methods Research Designs Twenty Years Later

quantitative and qualitative data-collection stages. Our definition reflected a design orientation and the current discussions that had occurred in the prior 10–15 years in the field. By 2018, we included more information about the key features of mixed methods research. We wrote: The researcher: • collects and analyzes both qualitative and quantitative data rigorously in response to research questions and hypotheses, • integrates (or mixes or combines) the two forms of data and their results, • organizes these procedures into specific research designs that provide the logic and procedures for conducting the study, and • frames these procedures within theory and philosophy (Creswell & Plano Clark, 2018, p. 5). In this definition, the core characteristics now incorporate the idea of specific mixed methods designs that include both quantitative and qualitative data, as well as their integration. The idea of designs had become a major feature of our thinking about mixed methods research. Since our 2018 writing, we feel that one more feature should come into the definition of mixed methods—the idea of metainferences. This means that while the researcher draws inferences from the analysis of both quantitative and qualitative data in

Collect, Analyze, and Interpret Quantitative Data to Address a Research Question

a mixed methods study, additional inferences or insights, metainferences—result from conclusions learned from the integration of the two databases. One could argue that these metainferences, or insights from combining the two databases, provide a unique feature of mixed methods research not found in other methodologies. Over the years, other researchers have discussed the importance of metainferences (e.g., Johnson et al., 2007; Teddlie & Tashakkori, 2009), but did not give it the prominence in a definition that we now hold for it. From our perspective, at its core, mixed methods consist of “mining” the data beyond the quantitative and qualitative analysis, and gaining insight through integration and metainferences. Using this thinking, it might be helpful to visualize the core characteristics of mixed methods research, and we developed Figure 2.1 to illustrate the interconnectedness of these elements. We see in this figure the collection and analysis of both quantitative and qualitative data, the integration of the two databases in a design, and the drawing of insights or metainferences from examining the integration. This focus on the central features of mixed methods, we feel, is ideal for beginning mixed methods students, as well as for international scholars new to the field. It has become a central feature in presentations for training programs (Creswell, 2021), and focusing on a small set of key features helps researchers conceptualize their mixed methods study.

Integrate the Data (merge, connect, build) In a Research Design and Set of Procedures

Collect, Analyze, and Interpret Qualitative Data to Address a Research Question

Draw Metainferences that Add Insight Beyond the Quantitative and Qualitative Data

Figure 2.1  A simplified presentation of the interconnection of four core components of mixed methods research Source: Original author created.

25

26

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Naming and Central Elements of Designs Although there is no consensus about the names or number of mixed methods designs, the types have evolved considerably to reflect a greater understanding of procedures. We have embraced a parsimonious set of designs at the core of mixed methods research, but have added how these core designs can be linked with additional frameworks or processes to form complex designs. In the 2003 chapter, we provided a table where we listed eight different classifications of designs beginning with one from 1989 (Greene et al., 1989) and ending with one from 1999 (Creswell, 1999). Over this ten-year period, we saw how the quantitative and qualitative databases in designs were timed differently, given higher or lower priority, integrated, and combined with theoretical perspectives. From this review, we identified and named six types of mixed methods designs: sequential explanatory design, sequential exploratory design, sequential transformative design, concurrent triangulation design, concurrent nested design, and concurrent transformative design. Twenty years later, the names and descriptions of mixed methods designs have changed considerably.

Naming of the designs

Looking at the names for mixed methods designs, we find that they have certainly changed over time to reflect changing perspectives about mixed methods research and how the designs can be communicated to audiences. These changes are particularly apparent in our reflections across three editions of our mixed methods book (Creswell & Plano Clark, 2007, 2011, 2018). We included tables in each edition that listed the growing number of classifications of types of mixed methods designs, their names, and their authors, with 15 different classifications listed in 2018. We must admit that currently there is no consensus about the types or the names of designs. With increased use of mixed methods research in empirical studies across disciplines, we do find that certain types of designs and names are becoming established as common nomenclature, and expect this trend to continue.

Specifying central elements of the designs

The changing names for mixed methods design types correspond with evolving thinking about the central elements of the mixed methods designs. Specifically, we have seen changes about the central elements that include:

• We have reduced the emphasis on timing for the designs. For example, the term “sequential” has been moved into a secondary position and we now talk about “explanatory sequential” designs. Likewise, “concurrent” has been eliminated and we now talk about “convergent” designs. These names reflect less the timing of two databases and more the integration procedures and intent. • The emphasis on priority, and designating it with a notation of uppercase letters and lowercase letters (e.g., QUAN or quan), has been downplayed. We believe that this de-emphasizes an unequal hierarchy of importance of one database over the other. • We have emphasized the importance of “intent” (or underlying logic) in different mixed methods designs, and ask researchers to focus on the reasons for why they are collecting both quantitative and qualitative data, and why integration is important for their study. • Theory has become an important feature for all mixed methods design applications. We encourage researchers to identify a substantively relevant theory or conceptual framework for their study, and then to specify how the theory relates to the design being used, and where it enters into the procedures of the design. • We have come to think of philosophy (e.g., transformative perspectives) as important, but separate from designs as each mixed methods design can be applied from different philosophical stances. We urge researchers to identify their philosophical assumptions or worldviews that inform their projects (e.g., epistemology, ontology), such as described in Shannon-Baker’s (2016) overview of mixed methods philosophies (pragmatism, transformative–emancipatory, dialectics, and critical realism).

Identifying core designs

In short, the way we see designs now bears little resemblance to our 2003 typology. Today, we see two major categories of mixed methods designs: core designs that include the collection of both quantitative and qualitative data and bring together the two databases, and complex designs that combine core designs with additional frameworks or processes. We feel that all mixed methods studies include one or more core designs, and researchers need to identify the core design(s) they are using. The three major core designs we advanced are: convergent designs (overall domains are

Revisiting Mixed Methods Research Designs Twenty Years Later

developed from merging the two databases), explanatory sequential designs (quantitative results are further explained qualitatively), and exploratory sequential designs (qualitative results are further tested or measured quantitatively). Although these three designs capture foundational logics found within mixed methods approaches, we learned over the years that core designs did not capture the entire spectrum of mixed methods projects, and we began to include the application of core designs within complex processes or frameworks.

Describing complex designs

Our thinking about complex designs developed through our consulting and trainings where many scholars noted that our core designs did not fully reflect the procedures they were using in their studies. Our emphasis on variants within core designs as well did not adequately address the scope of some studies. Then, a major step forward in our thinking about moving toward complex designs came from two books, both published in 2016. Plano Clark and Ivankova (2016) discussed advanced applications of mixed methods designs and suggested that core designs were often combined with additional approaches (i.e., other methodologies or frameworks). They conceptualized this combination as intersecting mixed methods with other approaches and provided examples such as mixed methods experiments, mixed methods case study, and mixed methods action research. Also in 2016, Nastasi and Hitchcock, in discussing their long-term evaluation project in Sri Lanka, suggested that some mixed methods projects were “complex” (p. 42) because the project involved multiple research phases, were conducted over several years, were supported with substantial funding, and included mixed methods core designs within different phases of the research. It was this last point that especially caught our attention. Perhaps there was an entire category of designs that moved beyond the core designs, but included them in the steps of a more “complex” research process. With these thoughts in mind, we then expanded our discussion of mixed methods designs to include complex designs such as mixed methods experimental or intervention trials, participatory research processes, multiple case studies, and evaluation approaches (Creswell & Plano Clark, 2018).

Depicting the design types with diagrams

Alongside the increasing complexity of mixed methods designs arose the problem of how to

27

discuss and visually present the complex designs for readers. This was true, as well, for our core designs, and over the years, the diagrams or pictures of types of mixed methods designs have changed considerably. In 2003, the idea of differentiating the mixed methods designs with diagrams was still rather new. We were inspired by Steckler et al.’s (1992) visual depictions of different models for combining quantitative and qualitative methods and incorporated our own diagrams of the six designs that we discussed in our 2003 chapter. Figure 2.2 provides examples of such diagrams spanning 1992 to 2018 for the convergent mixed methods design. Examining just these few examples highlights changes in diagrams used to depict the design types over time that reflect the changing perspectives about the central elements of the designs. Notably, there has been decreased attention to the quantitative and qualitative components and increased attention to the points of integration within the design types. In addition, we note that the diagrams increasingly used additional graphical elements such as different kinds of shapes and the use of shading when depicting the types of mixed methods designs.

Visual Representations of Designs for Mixed Methods Studies The use of diagrams for depicting the designs for mixed methods studies has become popular in study reports, even required in some publication venues. We now have much more elaborate diagrams and better pictures of our mixed methods procedures. Our 2003 chapter included diagrams to describe the procedures used in actual mixed methods studies in addition to the diagrams that depicted the different design types. The diagrams of the mixed methods designs in studies were simple drawings that included quantitative and qualitative data collection, data analysis, results, and interpretation. Since 2003, study diagrams have become common means for portraying mixed methods procedures, and they have become more detailed, and vary depending on the core and complex designs.

Expanding interest in using diagrams

We started developing diagrams for mixed methods study procedures during this 2003 time-period and shortly thereafter based on conversations we had with National Institutes of Health (NIH) program officers. These officers told us that they liked mixed methods studies but could not

28

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

1992 | Steckler et al.'s model of using qualitative and quantitative methods equally and parallel

QUALITATIVE

RESULTS

QUANTITATIVE

2003 | Creswell et al.'s visual of the concurrent triangulation design

+

QUAN QUAN Data Collection

QUAL QUAL Data Collection

QUAN Data Analysis

Data Results Compared

QUAL Data Analysis

2016 | Plano Clark and Ivankova's concurrent Quan + Qual mixed methods design logic Quan Strand Compare or relate Qual Strand

Draw inferences based on two sets of results

2018 | Creswell and Plano Clark's general diagram of the convergent design Quantitative Data Collection and Analysis Qualitative Data Collection and Analysis

Results merged and compared

Interpretation

Figure 2.2  Evolution of diagrams used to depict the convergent mixed methods design type over time Source: Original author created.

understand them because they were too complex given their multiple forms of data collection and data analysis. At this point, we began drawing diagrams of procedures. We knew that participant flow diagrams were available in the CONSORT guidelines for intervention trials (consort-statement.org) and that researchers often drew figures of the theories operating in studies. In qualitative research, figures of processes and themes had been popular renderings over the years. What

emerged from this early time period has been elaborate diagrams and their inclusion of specific detail to describe the study’s mixed methods designs, their encouragement in rigorous mixed methods studies, and their link to specific core and complex design types. Procedural diagrams are figures or drawings of the method procedures (the data collection, analysis, integration, and interpretation) in a mixed methods study. They have become a

Revisiting Mixed Methods Research Designs Twenty Years Later

centrepiece for mixed methods research that holds value for planning mixed methods studies, keeping projects on track with team members and stakeholders, for orienting graduate committees to the major features of a proposed mixed methods study, and for capturing in summary form the procedures of study for proposal developers and reviewers for federal and private foundation funding. Today, they are encouraged as one feature of a high-quality mixed methods publication. For example, they have been mentioned in the Journal of Mixed Methods Research editorials as part of rigorous criteria for a publication (Fetters & Molina-Azorin, 2019a, 2019b). They have also been emphasized in the “best practices” guidelines issued by the NIH, Office of Behavioral and Social Science (2011, 2018).

Increasingly detailed diagrams

From the rudimentary drawings of 2003, the field of mixed methods has moved toward increasingly sophisticated details to include in the diagrams. The diagrams are now linked to the type of mixed methods designs, with pictures that portray the different unfolding of steps in a project. For example, Ivankova et al. (2006) provided guidelines for developing diagrams and advanced a three-column picture of an explanatory sequential design that included (a) the flow of the quantitative, qualitative, and integrative stages; (b) the specific procedures that researchers engaged in at each stage; and (c) the outcomes or “products” at each stage. Furthermore, Morse and Niehaus (2009) introduced the use of a circle and arrow to indicate the point in a study where integration occurred, and Teddlie and Tashakkori (2009) used arrows with dashed lines to indicate interactions among quantitative and qualitative components of a mixed methods study. Across the many developments, the diagrams of mixed methods studies’ designs have incorporated new detail over the years, such as the inclusion of: • boxes for major quantitative and qualitative elements; • arrows that indicate the flow over time; • circles for places of decisions or interpretation; • a circle and/or arrow that indicates where integration occurs in the procedures; • study aims or questions that attach to each major phase in the study (quantitative, qualitative, and mixed methods aims); • a timeline with dates that can persuade graduate committees or funders of a successful completion of the project; • color coding or shading to distinguish the quantitative, qualitative, and integrative elements;

29

• linear (horizontal or vertical) layouts of procedures to indicate the flow of activities; • titles for the figures that mention the type of mixed method design and the general topic of the study; • a box for the theoretical framework being used in the study and where it contributes to the procedures; • the philosophical orientation (e.g., critical realism, pragmatism) that pervades the entire research steps; and • validity issues in procedures where the accuracy or inferences drawn might be challenged.

Core and complex design diagrams

The procedures of mixed methods studies using core designs may not be difficult to draw using the details listed above (e.g., Ivankova et  al., 2006). Audiences can be taken through steps in drawing diagrams. In 2016, Creswell made a presentation at the Mixed Methods International Research Association global conference on how the core designs might be drawn in a PowerPoint presentation. It started with boxes for quantitative and qualitative data collection and analysis; arrows added to show sequence; an interpretation circle to show where metainferences could be obtained; study aims for the quantitative, qualitative, and integration phases; a title that commented on design and topic; and finally, a timeline for the phases. We would now add to these features a box for integration to show where it occurs in the drawing. Strategies for drawing a diagram for a study using a complex mixed methods design have also progressed over time. A general approach for drafting such a diagram would be to first list the steps in the overall study process in which the core mixed methods designs will be embedded. For example, Ivankova (2015) presented a diagram organized by the steps typically taken in a participatory action research study; Nastasi and Hitchcock (2016) detailed their phases in a program evaluation; Plano Clark et al. (2013) showed the process phases for an experimental intervention study; and Swindle et al. (2017) identified the phases of an implementation science study. In all of these studies, the authors added information into their diagrams to show the use of quantitative data, qualitative data, or both in the phases of the study. Consequently, as suggested by Creswell (2022), the design of a diagram for a complex design can start with listing the steps in the framework or process (e.g., participatory action research study or a program evaluation), indicate at what

30

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

steps the researcher has the opportunity to gather both quantitative and qualitative data (for a mixed methods core study), and then show in the diagram (or in additional diagrams) the type(s) of core mixed methods study being embedded in different process steps.

Integration in Mixed Methods Designs For years, integration has been a misunderstood subject in the field of mixed methods designs. Now greater clarity has emerged through parsing the concept and its use in the design and implementation of a study. When we wrote our chapter about mixed methods designs in 2003, the stage of integration was one of the four criteria we advanced for distinguishing different types of mixed method designs. Our focus on integration then detailed where integration occurred in the research process, such as in the research questions, data collection, data analysis, and interpretation. The discussion essentially kept our commentary focused on the quantitative or qualitative data, and not their combination or integration. Since that time, the understanding and emphasis of integration has evolved considerably.

Calls for understanding integration

Shortly after our 2003 chapter, Bryman (2006) and O’Cathain et  al. (2007) raised the alarm about how little integration was understood and used in mixed methods studies. These writers came from England, and they had reviewed published mixed methods projects and found little inclusion of the integration concept. Since that time, extensive literature has been published examining the nature of integration in close details. In 2020, Guetterman et  al. summarized sixteen journal articles and three editorials that had addressed mixed methods integration. They reviewed articles submitted between 2015 to 2020 in the Journal of Mixed Methods Research as well as earlier writings stemming back to 2007. It has been said that a major topic in the methodology of mixed methods research is the component of integration, a hallmark of the field (Fetters & Freshwater, 2015). Integration is an essential element of mixed methods research because it is the process by which the researcher brings together the quantitative and qualitative data in a study (or in a program of study) allowing for the generation of metainferences (Creswell, 2022).

Parsing the concept of integration

We can now pinpoint numerous issues that authors have addressed about integration in mixed methods designs over the last ten years. These efforts fall into several categories: the scope of integration, strategies and representations (joint displays) of integration within mixed methods studies, and descriptions of the “intent” and “procedures” of integration in specific mixed methods designs. On the scope issue, important questions have been raised about the nature of integration. For example, does integration apply to the data collection, analysis, and interpretation only, or is it much broader than the methods, as applied to the “whole research process” (Akerblad et al., 2021, p. 168) or executed at every phase of the study (Creamer, 2018)? One way is to extend integration beyond the methods to include the social process or context (e.g., the stakeholders) surrounding research and the desired outcomes of integration, as proposed by Lynam et  al. (2020). Fetters and Molina-Azorin (2017) also echoed a larger role for integration beyond methods, when they suggested that the combination of qualitative and quantitative research created “a new whole or a more holistic understanding than achieved by either alone” (p. 293). Moseholm and Fetters (2017) discussed integration as operating at different levels, including the philosophical, design, methods, and interpretation levels. Our stance is that the broader conceptualization of integration does exist beyond the methods, but its highlighting does little to provide guidance for the actual conduct of a mixed methods study. Our approach, therefore, focuses on procedures for integration at the methods level within specific mixed methods designs. Adopting this line of thinking, we can see integration not only as an abstract concept, but also as one tied directly in the types of mixed methods designs and the procedures used. Fetters et al. (2013) described four overall integration approaches of merging (combining two sets of data/results), connecting (using one type of data/result to inform sampling of the other type of data), building (using one type of data/result to inform the collection approach for the other type of data), and embedding (linking data collection and analysis at multiple points). Each of these occurs at a different point within mixed methods studies and, as such, we find that this conceptualization aligns well with the different logics of the core mixed methods designs. Increasing attention is being placed on strategies and representations of integration within mixed method studies. For example, Bazeley (2018) advanced a book describing integrative analyses strategies that span approaches including

Revisiting Mixed Methods Research Designs Twenty Years Later

sequential, complementary, linked, and data transformation strategies that occur during the analysis stage of mixed methods projects. Johnson et  al. (2019) discussed specific integration procedures of listing, matching, checking, and pillar building, in which quantitative data and categories are brought together with qualitative codes and themes. A strategy becoming common is the development and use of joint displays, which are tables and graphs in which the quantitative and qualitative data are arrayed together for combined interpretation (Guetterman et al., 2015). For example, quantitative and qualitative results can be compared by domain and insight drawn from interpreting the fit between the data through concordance, expansion, complementarity, and discordance (Fetters, 2020). Joint displays are a useful analytic strategy as well as a good way to represent integration within mixed methods publications, and they are a creative element in mixed methods studies. Not only do joint displays array the quantitative and qualitative information in a table, they may also be represented in ever-expanding circles (Bustamante, 2017) or in images or pictures combined with numbers (Peroff et al., 2020). Finally, integration needs to be described in the reports of mixed methods studies. One helpful way to think about writing about integration in a procedural sense is to consider its “intent” and its “procedures” within the types of mixed methods designs. Both of these aspects have been given greater emphasis in our discussion about types of designs (Creswell & Plano Clark, 2018). As shown in Table 2.1, Creswell (2022) presented “intent” and “procedures” for core mixed methods

31

designs and for complex designs, and highlighted the words that researchers might use in an integration statement in their study, building on the integration approaches described by Fetters et al. (2013). Statements such as those in Table 2.1 help to bring integration to the forefront in mixed methods studies, both for the authors and the readers of the reports, and make the fit between integration and the mixed methods design transparent.

CHALLENGES TO MIXED METHODS DESIGNS As the understanding of mixed methods designs has developed, so too has our recognition of the associated challenges. Unquestionably, with the different types of designs and names, the language for designs remains problematic. Should designs that combine with additional frameworks or processes be called advanced designs (Plano Clark & Ivankova, 2016), complex designs (Creswell & Plano Clark, 2018), integrated designs (Creamer, 2018), scaffolded designs (Fetters, 2020), or something else? When the term “explanatory” is used in an explanatory sequential design, what exactly is being explained? When integration strategies are being presented for the designs, do we use the full array of integration terms such as comparing, matching, expanding, diffracting, building, and embedding (Fetters, 2020) or a much smaller set and define each term? These confusing and often overlapping terms in many

Table 2.1  Intent, procedures and wording for mixed methods integration Type of design

Intent of integration

Procedures for integration

Convergent design

Compare or match the two results to confirm the results or to examine discrepancies between them. Explain the surprising, unusual, or notable results with qualitative data.

Merge the data by placing the results sideby-side in a table (e.g., joint display).

Explanatory sequential design

Exploratory sequential design

Explore with the qualitative data to enhance the cultural specificity of the quantitative assessment.

Complex designs

Enhance the framework or process by adding qualitative or quantitative data or both.

Source: Reprinted and adapted with permission from Creswell (2022). Sage.

Connect the qualitative data collection to the quantitative results to follow-up. Merge the two sets of results for explanation. Build or expand the quantitative assessment by incorporating culturally specific qualitative findings Merge the two sets of results for enhanced cultural interpretation. Embed the qualitative and quantitative data into a framework or process.

32

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

aspects of mixed methods research make it difficult to learn and to use, despite the many glossaries found in mixed methods textbooks. Another continuing challenge is how best to choose a mixed methods design type. Options for consideration of answering this challenge range from using a design that best matches the question, selecting a design that reflects the intent of collecting both quantitative and qualitative data and integrating it, and identifying a design popular in the researcher’s field of study, or one embraced by graduate committee supervisors (Creswell & Plano Clark, 2018). The flexibility and complexity of mixed methods design options are both an advantage and a complication of using mixed methods. Until greater clarity (and perhaps consensus) exists about the design options available and their relative methodological strengths, it will continue to be a challenge for researchers to decide which design best fits their study. Writers have challenged discussions about designs from the standpoint of whether the designs give priority to quantitative research and minimize the importance of qualitative research. For example, Morse and Niehaus (2009) contend that there is a “devaluation of the qualitative component … or the devaluation of anything less than experimental designs” (p. 19). This devaluation might come from seeing some mixed methods designs as an “add-on” to quantitative research with the use of qualitative to supplement traditional quantitative approaches. We do not disagree that mixed methods projects tend to prioritize either quantitative or qualitative based on the researchers’ perspectives and purposes. However, we find that the different designs can be either qualitatively or quantitatively driven depending on the research context. We also feel that the qualitative component can be valued and play an important role even when included within a quantitatively oriented project, such as a mixed methods experiment. Perhaps the most substantive challenge to mixed methods designs comes from a group of scholars in the mixed methods community who question whether choosing a design type is, in fact, a useful way to think about mixed methods research (for an in-depth discussion, see also Chapter 3). They suggest alternative conceptualizations of mixed methods design. Maxwell and Loomis (2003) felt that a “typology” design perspective cast a narrow perspective on the research process. Using systems theory, they proposed an interactive design approach that considers the purpose, the conceptual framework, the questions, the methods and validity, with the research questions positioned at the centre of the interaction. Hall and Howard (2008) advanced the “synergistic” approach as an alternative

between typological and system approaches. In their approach, they situated four core principles between quantitative and qualitative foundations: synergy, the equal value attributed to quantitative and qualitative research, an honouring of ideological differences between the two research approaches, and a consideration of the relationships between researchers on a collaborative team and their shared perspectives in the design of a study. Guest (2013) argued that the variety of typologies available were inadequate to capture the complexity of real-world research projects and mixed methods designs. Consequently, he said that the description of research scenarios would be improved by shifting the focus from design types to points of interface, where the quantitative and qualitative datasets mix or connect. Thus, he focused attention on integration in mixed methods research, and highlighted both timing and purpose for integration, as key elements for describing a study’s mixed methods design. More recently, Poth (2018) focused on the need to respond to the conditions surrounding a mixed methods study and described an alternative approach to mixed methods design from a lens of complexity theory and advocated the use of adaptive practices that respond to a study’s unique conditions. Creamer (2018) advanced a “fully integrated approach” (p. 12) to mixed methods design that emphasizes weaving quantitative and qualitative thinking throughout each of five stages of a study (planning, data collection, sampling, analysis and inferences). While acknowledging the ongoing challenges, we still adhere to the value of a typology approach and its rationale as a tool to help researchers design studies because the approach provides a common language, offers structure for procedures, gives legitimacy to the field, and provides a pedagogical tool to help researchers, as argued by Teddlie and Tashakkori (2009). The fact that researchers have historically collected both quantitative and qualitative data and perhaps integrated the databases, without using designs (Maxwell & Loomis, 2003), speaks to us as to how the field of mixed methods has evolved. Moreover, over the years, researchers have relied on designs, and are familiar with them in both quantitative and qualitative research. To continue the use of designs in mixed methods creates an approach to research familiar to investigators. In response to concerns about the complexity of mixed methods designs, we feel that the identification of complex designs begins to address the issue of the designs found within large projects, and we continue to highlight the importance of integration within the designs. This being said, to keep the field open to alternative approaches is useful, and the future needs

Revisiting Mixed Methods Research Designs Twenty Years Later

to embrace multiple ways of engaging in mixed methods research.

FUTURE TRENDS IN MIXED METHODS DESIGNS This retrospective examination of mixed methods designs contributes to the field of mixed methods in several ways. It highlights the centrality of designs within the field of mixed methods. Furthermore, it chronicles how designs have evolved and continue to change over the years in response to changing perspectives and advanced understandings. Along with the developments, it highlights several controversies that exist around design-thinking in mixed methods. The chapter also assembled key references from the design literature that readers can consider for citation in future studies. Turning to the future, we imagine the field of mixed methods building on several trends already underway in design thinking. From the appearance of “set” designs reported in empirical studies and in mixed methods books, the flexibility of “adapted” designs, especially those applied in multiphase and longitudinal projects will likely receive more attention moving forward. From the cluster of identified core designs and complex designs, “families” of designs will likely emerge to accommodate the many variants that researchers use in practice (Creswell & Plano Clark, 2018; Tashakkori et al., 2021). From the initial exploration of complex designs will develop an explosion of new types that extend the reach of mixed methods to approaches already starting to appear in the mixed methods literature, such as social network analysis, photovoice, intervention studies, implementation science, racial/critical theory, indigenous methodologies, and the like. In addition to broadening the types of designs, we anticipate there will be deeper considerations given to the nuanced details of specific designs, such as developing approaches to validity for types of designs and considerations for ethics within the designs’ procedures. From researchers’ creative efforts to develop joint displays will come new visuals that link qualitative images and social media data with quantitative numeric information to support metainferences. We also anticipate joint displays that take advantage of modern technologies such as by incorporating non-static graphical presentation of information (see also Chapter 22, this volume). From the current diagrams linked to the methods used in studies, we will likely see the continuing

33

sophistication of detail tailored to stakeholders, policy decision-makers, and the ever-expanding international use of mixed methods. We will see mixed methods adaptations of diagrams already used in other methodologies, such as mixed methods versions of CONSORT flow diagrams for experiments (www.consort-statement.org/) and PRISMA diagrams in literature syntheses (www. prisma-statement.org/). We also envision that the field will focus attention on applications of mixed methods to pressing societal problems such as pandemics, natural disasters, and the migration of the world’s populations. A case in point would be the special issue on responses to COVID-19 in the Journal of Mixed Methods Research (Fetters & Molina-Azorin, 2021). We expect that mixed methods designs will support new ways to respond to immediate social problems, while at the same time expanding thinking about long-term research agendas, such as the process of developing, testing and implementing new interventions. In short, when we wrote the chapter on “advanced mixed methods designs” in 2003, we could not have envisioned the tremendous changes that would occur in the field in the subsequent 20 years. As we look back, it has been a remarkable path of steady growth in thinking about designs, a growth that we have been privileged to contribute to. We eagerly anticipate the new perspectives, creative thinking and advanced procedures that scholars will bring to the understanding and application of mixed methods designs in the years ahead, starting with the chapters in this important and timely volume.

WHAT TO READ NEXT To build on the ideas discussed in this chapter, we recommend that readers examine the following sources for more information. Creswell, J. W. and Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.) Sage.

In this third edition, we discuss how our typology of mixed methods design types has changed over time and provide details about the intent, procedures, and considerations associated with prominent core and complex mixed methods designs. Guest, G. (2013). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7(2), 141–151. https:// doi.org/10.1177/1558689812461179

34

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

In this article, Guest discusses both advantages and disadvantages of design types for mixed methods research, while offering his own perspective of how investigators should describe their complex mixed methods studies. Section 3: Expanding Design Approaches (this volume).

The chapter authors present a variety of avantgarde approaches that join mixed methods with emergent methodologies and/or culturally sensitive perspectives to form complex mixed methods designs.

REFERENCES Akerblad, L., Seppänen-Järvelä, R., & Haapakoski, K. (2021). Integrative strategies in mixed methods research. Journal of Mixed Methods Research, 15(2), 152–170. https://doi.org/10.1177/1558689820957125 American Psychological Association (2020). Publication manual of the American Psychological Association (7th ed.). American Psychological Association. Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage. Bryman, A. (1988). Quantity and quality in social research. Routledge. Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6(1), 97–113. https://doi.org/10.1177/ 1468794106058877 Bustamante, C. (2017). TPACK and teachers of Spanish: Development of a theory-based joint display in a mixed methods research case study. Journal of Mixed Methods Research, 13(2), 163–178. https:// doi.org/10.1177/1558689817712119 Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Houghton-Mifflin. Creamer, E. (2018). An introduction to fully integrated mixed methods research. Sage. Creswell, J. W. (1994). Research design: Qualitative and quantitative approaches. Sage. Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five approaches. Sage. Creswell, J. W. (1999). Mixed method research: Introduction and application. In T. Cijek (Ed.), Handbook of educational policy (pp. 455–472). Academic Press. Creswell, J. W. (2016, August 6). Application of recent innovations in mixed methods research procedures: How to develop a diagram of your mixed methods design. [Conference presentation].

MMIRA 2016 International Conference, Durham, England. Creswell, J. W. (2021). Core and complex designs. An online webinar presentation for scholars at the NIH, Mixed Methods Research Training Program. Creswell, J. W. (2022). A concise introduction to mixed methods research (2nd ed.). Sage. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Sage. Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Sage. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Creswell, J. W., Plano Clark, V. L., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 209–240). Sage. Fetters, M. D. (2020). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Sage. Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs— principles and practices. Health Services Research, 48(6), 2134–2156. https://doi.org/10.1111/ 1475-6773.12117 Fetters, M. D., & Freshwater, D. (2015). Publishing a methodological mixed methods research article. Journal of Mixed Methods Research, 9(3), 203– 213. https://doi.org/10.1177/1558689817712119 Fetters, M., & Molina-Azorin, J. (2017). The Journal of Mixed Methods Research starts a new decade: The mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11(3), 291–307. https://doi.org/10.1177/ 1558689817714066 Fetters, M. D., & Molina-Azorin, J. F. (2019a). A checklist of mixed methods elements in a submission for advancing the methodology of mixed methods research. Journal of Mixed Methods Research, 13(4), 414–423. https://doi.org/10.1177/ 1558689819875832 Fetters, M. D., & Molina-Azorin, J. F. (2019b). New requirements to include the methodological contribution in articles published in the Journal of Mixed Methods Research. Journal of Mixed Methods Research, 13(2), 138–142. https://doi. org/10.1177/1558689819834753 Fetters, M. D., & Molina-Azorin, J. F. (2021). Special issue on COVID-19 and novel mixed methods methodological approaches during catastrophic social changes. Journal of Mixed Methods Research, 15(3), 295–303. https://doi.org/10.1177/ 15586898211029100

Revisiting Mixed Methods Research Designs Twenty Years Later

Greene, J. C., & Caracelli, V. J. (Eds.). (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms: New directions for evaluation, 74. Jossey-Bass. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https:// doi.org/10.3102/01623737011003255 Guest, G. (2013). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7(2), 141–151. https:// doi.org/10.1177/1558689812461179 Guetterman, T. C, Fetters, M. D., & Creswell, J. W. (2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm.1865 Guetterman, T. C., Molina-Azorin, J. F., & Fetters, M. D. (2020). Virtual special issue on “integration in mixed methods research.” Journal of Mixed Methods Research, 14(4), 430–435. https://doi. org/10.1177/1558689820956401 Hall, B., & Howard, K. (2008). A synergistic approach: Conducting mixed methods research with typological and systemic design considerations. Journal of Mixed Methods Research, 2(3), 248–269. https://doi.org/10.1177/1558689808314622 Hossler, D., & Vesper, N. (1993). An exploratory study of the factors associated with parental savings for postsecondary education. Journal of Higher Education, 64(2), 140–165. https://doi.org/10.1080/0 0221546.1993.11778420 Ivankova, N. V. (2015). Mixed methods applications in action research. Sage. Ivankova, N. V., Creswell, J. W., & Stick, S. (2006). Using mixed methods sequential explanatory design: From theory to practice. Field Methods, 18(1), 3–20. https://doi.org/10.1177/1525822X05282260 Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/1558689806298224 Johnson, R. E., Grove, A. L., & Clarke, A. (2019). Pillar integration process: A joint display technique to integrate data in mixed methods research. Journal of Mixed Methods Research, 13(3), 301–320. https://doi.org/10.1177/1558689817743108 Levitt, H. M., Bamberg, M., Creswell, J. W., Frost, D. M., Josselson, R. & Suárez-Orozco, C. (2018). Journal article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 26–46. http://dx.doi.org/10.1037/amp0000151 Lynam, T., Damayanti, R., Titaley, C. R., Suharno, N., Bradley, M., & Krentel, A. (2020). Reframing

35

integration for mixed methods research. Journal of Mixed Methods Research 14(3), 336–357. https:// doi.org/10.1177/1558689819879352 Maxwell, J. A., & Loomis, D. M. (2003). Mixed methods design: An alternative approach. In A. Tashakkori & C. Teddlie (Eds.), Handbook of Mixed Methods in Social & Behavioral Research (pp. 241–271). Sage. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: A sourcebook of new methods. Sage. Molina-Azorin, J. F., & Fetters, M. D. (2022). Books on mixed methods research: A window on the growth in number and diversity. Journal of Mixed Methods Research, 16(1), 8–16. https://doi. org/10.1177/15586898211068208 Morgan, D. L. (1998). Practical strategies for combining qualitative and quantitative methods: Applications to health research. Qualitative Health Research, 8(3), 362–376. https://doi.org/ 10.1177/104973239800800307 Morse, J. M. (1991). Approaches to qualitative– quantitative methodological triangulation. Nursing Research, 40, 120–123. Morse, J. M., & Niehaus, L. (2009). Mixed methods design: Principles and procedures. Left Coast Press. Moseholm, E., & Fetters, M. D. (2017). Conceptual models to guide integration in mixed methods convergent studies. Methodological Innovations, 10(2). https://doi.org/10.1177/2059799117703118 Nastasi, B. K., & Hitchcock, J. (2016). Mixed methods research and culture-specific interventions: Program design and evaluation. Sage. National Institutes of Health Office of Behavioral and Social Sciences Research (2011, 2018). Best practices for mixed methods research in the health sciences. https://obssr.od.nih.gov/research-resources/ mixed-methods-research O’Cathain, A., Murphy, E., & Nicholl, J. (2007). Integration and publications as indicators of “yield” from mixed methods studies. Journal of Mixed Methods Research, 1(2), 147–163. https://doi. org/10.1177/1558689806299094 Onwuegbuzie, A. J. (2012). Putting the MIXED back into quantitative and qualitative research in educational research and beyond: Moving toward the radical middle. International Journal of Multiple Research Approaches, 6(3), 192–219. https://doi. org/10.5172/mra.2012.6.3.192 Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Sage. Peroff, D. M., Morais, D. B., Seekamp, E., Sills, E., & Wallace, T. (2020). Assessing residents’ place attachment to the Guatemalan Maya landscape through mixed methods photo elicitation. Journal of Mixed Methods Research, 14(3), 379–402. https://doi.org/10.1177/1558689819845800

36

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Plano Clark, V. L., & Ivankova, N. V. (2016). Mixed methods research: A guide to the field. Sage. Plano Clark, V. L., Schumacher, K., West, C., Edington, J., Dunn, L.B., Harzstartk, A., & Miaskowski, C. (2013). Practices for embedding an interpretive qualitative approach within a randomized clinical trial. Journal of Mixed Methods Research, 7(3), 219–243. https://doi.org/ 10.1177/1558689812474372 Poth, C. N. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Shannon-Baker, P. (2016). Making paradigms meaningful in mixed methods research. Journal of Mixed Methods Research, 10(4), 319–334. https:// doi.org/10.1177/1558689815575861 Steckler, A., McLeroy, K. R., Goodman, R. M., Bird, S. T., & McCormick, L. (1992). Toward integrating qualitative and quantitative methods: An introduction. Health Education Quarterly, 19(1), 1–8. https://doi.org/10.1177/109019819201900101

Swindle, T., Johnson, S. L., Whiteside-Mansell, L., & Curran, G. M. (2017). A mixed methods protocol for developing and testing implementation strategies for evidence-based obesity prevention in childcare: A cluster randomized hybrid type III trial. Implementation Science, 12(90). https://doi. org/10.1186/s13012-017-0624-6 Tashakkori, A., Johnson, R. B., & Teddlie, C. (2021). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences (2nd ed.). Sage. Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Sage. Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social & behavioral research. Sage. Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage.

3 Mixed Methods Design in Historical Perspective: Implications for Researchers Joseph A. Maxwell

INTRODUCTION This chapter discusses the development of mixed methods design in historical perspective. (For a review of more recent works, see Molina-Azorin & Fetters, 2022.) I review the different views of design (implicit as well as explicit) that are present in this range of studies, and how these have changed over time. I conclude with three main implications for mixed methods researchers, dealing with the role of paradigms in research design, the conception of design itself, and the specific strengths, limitations and complementarities of qualitative and quantitative approaches. Typically, “design” in recent, explicitly identified “mixed methods research” has been understood typologically, as a prior choice from an array of possible design types (e.g., Creswell & Plano Clark, 2018, Chapter 3; Mertens, 2018, Chapter 2; Poth, 2018; Teddlie & Tashakkori, 2006; Tashakkori et  al., 2021, Chapter 5). This conception of design is common in experimental and quasi-experimental research (e.g., Cook & Campbell, 1979), but is rarely found in qualitative research, in which different designs are often conceptualized as based on the epistemological and theoretical views held by the researcher; these views are assumed to shape the kinds of questions

and methods employed. Denzin and Lincoln (2000, pp. 18–23) described a number of “interpretive paradigms” (postpositivist, constructivist, critical, etc.) that shape the substantive theories, methods and presentation strategies of each type of study. Similarly, Creswell and Poth (2017) identified five approaches to qualitative research, and stated that the design of a study is related to the specific approach taken (pp. 1–2). However, neither of these works identify design types in the sense that these are found in experimental research. In this chapter, I define “mixed methods design”, more broadly than is typical in this field, in three ways. First, I identify as “mixed methods” all research that integrates (rather than simply employing separately) quantitative and qualitative methods or approaches. This applies regardless of how these approaches are labelled, and even if the researchers don’t explicitly identify this as integration or “mixed methods”. In addition, simply reporting numbers in a qualitative study, or data from open-ended interviews that are only analyzed quantitatively, isn’t generally considered mixed methods research (Maxwell, 2010). Mixed methods research is also often understood as combining, not just qualitative and quantitative methods and data, but also the assumptions and “mental models” of both approaches (Greene, 2007).

38

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Second, I consider “design” as referring not just to the prior plan for the research, but also to the actual design of the research “on the ground”, as well as to changes in the researchers’ conceptualization of the study as this evolves (Maxwell, 2013). This is based on the wider definition of design in everyday use: “an underlying scheme that governs functioning, developing, or unfolding” and “the arrangement of elements or details in a product or work of art” (Webster’s Ninth New Collegiate Dictionary, 1984). The “on the ground” design of a study can be more complex, and can more closely integrate qualitative and quantitative elements of the study than an author’s description, or an initial reading of a research report, would suggest. Third, I see the design of a study as involving multiple components of a study, including the goals, conceptual framework, research questions, methods, and validity and generalizability issues, and their interaction (Maxwell, 2013; Maxwell et  al., 2015; Maxwell & Loomis, 2003). These components clearly interact during the development and evolution of a mixed methods study, rather than proceeding in a linear fashion; the integration of quantitative and qualitative concepts, and practices may differ for each of these components, and may change during the course of the study. Seen in these ways, mixed methods design has a much longer history than has usually been acknowledged in the mixed methods literature; it has also been practised more widely than this literature has generally recognized (Maxwell, 2016, 2018). Unfortunately, for much of this history, we have no evidence for how the researchers thought about combining what we now call “quantitative” and “qualitative” methods and approaches. In addition, the terminology used for research methods has changed over time; for example, the current uses of the terms “quantitative” and “qualitative” in the social sciences are relatively recent and differ somewhat between social science fields. Even recent studies outside the “mixed methods” community rarely provide much information on how the researchers conceptualized combining the two approaches. In this chapter, therefore, I will attempt to reconstruct the researchers’ thinking about design from the existing evidence of their work. In what follows, I review the history of mixed methods research with respect to research design, both for explicit statements (which are rare), and for the conceptions of design that are implicit in the works described. I conclude by drawing conclusions from this history for the role of paradigms in design, for the conception of design itself, and

for the complementarity and integration of qualitative and quantitative approaches.

EARLY HISTORY OF DESIGN INTEGRATION Although this chapter will focus on the social, behavioural and health sciences, the earliest examples of research that combined qualitative and quantitative methods occurred in the natural sciences (Maxwell, 2016). Babylonian astronomers combined observational description of the planets’ motions and colours with mathematical calculations of their movements as early as 1000 BC (Heath, 1932/1991, pp. xvii–xviii), but the clay tablets describing this provide no insights into how they conceptualized this combination; Greek astronomy further developed this practice (Heath, 1932/1991). Similarly, Aristotle’s investigations in biology, in the fourth century BC, involved counting or measuring many features of diverse species, as well as describing these, and classifying animals into types on this basis (Wikipedia, n.d.). He did this “by noting that they have many general differences that vary in measurable ways—by the more and less, as he puts it” (Aristotle’s biology, 2017). However, although these studies clearly involved mixed methods design in the broad sense described above, Aristotle apparently wrote nothing about how he conceptualized this combination. The integration of visual description and quantitative measurement was further developed in astronomy with the telescopic observations by Galileo in the 1600s, and in geology by Lyell in the 1800s. (For a more detailed discussion, see Maxwell, 2016.) These studies involved an intentional and “on the ground” design that combined what we now call qualitative and quantitative methods. Although the qualitative descriptions in these examples from the physical sciences lack a key feature of much qualitative research in the social sciences—a focus on meaning—they are clearly qualitative in other senses. In the natural sciences, the incorporation of meaning, intention and other such “mental” phenomena appears later, in the study of the behaviour of non-human animals. Charles Darwin’s The Expression of the Emotions in Man and Animals (1872) integrated detailed descriptive observations and experimental investigation (Jabr, 2010), and made a major contribution to the development of ethology (the study of animal behaviour and thought) as a subfield of biology, one that has continually

MIXED METHODS DESIGN IN HISTORICAL PERSPECTIVE

integrated qualitative and quantitative methods (de Waal, 2016; Maxwell, 2016). In public health, the work of Snow on the cause of cholera in the 1850s relied on both observational fieldwork and quantification of the number of cases of cholera in neighbourhoods with different water sources (Freedman, 1991/2010, 2008). Quantitative and qualitative methods were also combined in the 1850s by Le Play in his studies of poverty in families in Europe (Zeisel, 1933/1971, pp. 109–112); this was continued by both Charles Booth and Jane Addams in their studies of social problems in the late 1800s. None of these researchers explicitly addressed the design of their research, but the “design-in-use” of these studies clearly involved the integration of quantitative and qualitative methods and data to reach their conclusions. What stands out from this work is that combining what we now call “quantitative” and “qualitative” concepts and methods was simply not seen as problematic in any way, to the extent that the two were not even clearly distinguished. It also seems that the concept of “research design” was also not explicitly articulated.

THE EMERGENCE OF EXPLICIT DISCUSSION OF INTEGRATION OF METHODS IN DESIGN To my knowledge, the first intentional and explicit integration of quantitative and qualitative methods in an empirical field study in the social sciences was W. E. B. DuBois’s The Philadelphia Negro (1899). DuBois stated that even “the best available methods of sociological research … are liable to error from the seemingly ineradicable faults of the statistical method, to even greater error from the methods of general observation’’ (pp. 2–3), and argued that “[t]he use of both of these methods which has been attempted in this study may perhaps have corrected to some extent the errors of each’’ (p. 3). The book contains many numerical tables, interspersed with observations, quotes from interviews, and excerpts from documents, clearly showing that the design of the study integrated the two approaches. Unfortunately, this work was ignored by later mixed methods researchers; I have never seen it cited in the self-identified “mixed methods” literature. As Platt (1996) noted, [DuBois] was black, and his race meant that he could not hope for a job in a research university;

39

thus he could not have the opportunity to train research students who would carry his legacy to the mainstream of white sociology. (p. 247)

Also largely ignored has been Max Weber’s insistence that the social surveys of the Verein für Sozialpolitik, in the late nineteenth century, address the entire pattern of life of the workers, subjective and cultural as well as objective (Zeisel, 1933/1971, p. 119), although Zeisel criticized Weber’s work as not adequately addressing the subjective/cultural side of this approach. During the 1920s and 1930s, a number of classic studies (Roethlisberger & Dickson, 1939; Lynd & Lynd, 1929; Jahoda et  al., 1933/1971; Warner & Lunt, 1941) were conducted that combined qualitative and quantitative methods, although not using these terms. These works have been mentioned in the mixed methods literature, but rarely analyzed for how the authors did this. The most explicit discussion of how the qualitative and quantitative methods were integrated in these works is in the Jahoda et al. study of unemployment (1933/1971, pp. 1–10). The researchers stated that “there is a gap between the bare figures of official statistics and the literary accounts .… The purpose of our study of the Austrian village, Marienthal, is to bridge this gap” (p. 1). They argued that “we have tried to build up a comprehensive picture of life in Marienthal, while at the same time accommodating complex psychological situations within an objective framework that is supported by relevant statistics” (p. 2). The design of the study was not further described, although they noted that some planned activities had to be abandoned, that new insights emerged during data collection, and that most of the analysis took place after the data had been collected. Zeisel concluded his review of earlier work up to the 1930s by stating: “The task of integration lies still ahead” (p. 125). An innovative example of incorporating quantitative concepts and methods in a qualitative design is Margaret’s Mothers of the South: Portraiture of the White Tenant Farm woman (1939/1996; see Maxwell, 2016, for a more detailed discussion). Hagood used statistical data in selecting her sample of farms and women to ensure representativeness, and in comparing her results with those from a separate sample in the Deep South, but she also used statistical concepts to analyze her qualitative data: We have tried to utilize case material to afford a richer sort of description than quantitative measures can give and yet to avoid the superficial, stereotyped, sentimental, “case study” .… In order to analyze and present this material in a more

40

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

scientific way than case study material is usually treated, we have used the two statistical concepts best suited to material for which no measures have been devised—the mode and the range of variation. These two measures, one of central tendency and the other of dispersion, … have the advantage of indicating for qualitative material the features that have the most meaning in everyday thinking—the type, or most usual, and the limits of the group under investigation in a particular trait. (Hagood, 1939/1996, pp. 228–229)

These two statistical concepts are repeatedly used in the presentation of the case material. This integration of quantitative concepts in an otherwise largely qualitative study is, to my knowledge, unique, and has had no influence on the development of mixed methods design. Existing typologies of mixed method research designs provide no insight into, or guidance for, this sort of integration, which began with the conceptual framework of the study and influenced its research questions, data analysis and validity strategies. This period saw the development of an explicit distinction between quantitative and qualitative methods, although not using these terms, and with less clear articulation of the differences between these than was developed later. It is also clear that these researchers saw no fundamental incompatibility between the two, and indeed identified definite advantages to combining the approaches.

FURTHER DEVELOPMENT OF DESIGN INTEGRATION IN THE MID-20TH CENTURY After around 1940, there was a decline in explicit discussion of the integration of qualitative and quantitative methods in most social science fields (anthropology was an exception, as discussed below). However, the actual integration of both approaches continued; Paul Lazarsfeld, in his Foreword to the American reissue of the Marienthal study (1971), stated that “The combination of quantification and interpretive analysis of qualitative material is today in the forefront of the research fraternity’s interest” (p. xxxvi). This period included studies that have been recognized as “classics” in particular fields (Becker et al., 1961; Blau, 1963; Festinger et al., 1956; Milgram, 1974).1 All of these studies involved the close integration of qualitative and quantitative concepts and methods, but have almost never been discussed in the self-identified “mixed methods”

literature, let alone in this literature’s treatment of “research design”. The most explicit discussion of “design” in the cited works from this period is by Becker et al. (1961). In their chapter titled “Design of the study”, they stated at the outset that In one sense, our study had no design. That is, we had no well-worked-out set of hypotheses to be tested, no data-gathering instruments purposely designed to secure information relevant to these hypotheses, no set of analytic procedures specified in advance. Insofar as the term “design” implies these features of elaborate prior planning, our study had none. If we take the idea of design in a larger and looser sense, using it to identify those elements of order, system, and consistency our procedures did exhibit, our study had a design. We can say what this was by describing our original view of the problem, our theoretical and methodological commitments, and the way these affected our research and were affected by it as we proceeded. (p. 17)

Many of the key concepts of the design were only developed during the research. This description closely matches the concept of interactive design described above; although it doesn’t address the specifics of this interaction, their account clearly describes how the authors’ assumptions and methods changed during the study. Blau (1963), in the second edition of his book, added a “methodological epilogue” dealing with the design and methods of his study. The “initial design” (pp. 271–274) involved both “broad problems to be explored” and formulating specific hypotheses, some of which were later abandoned or modified during the research. He “decided to use three basic methods—direct observation, interviewing, and analysis of official records— and to employ various quantitative and qualitative techniques under each” (p. 271). Many of the components of my “interactive design” model (theories, research questions, methods, validity concerns) changed during the course of the study as a result of the interaction of these components. Milgram’s “Obedience to authority” (1974) is the most explicit of these works in describing the integration of the experimental/quantitative and qualitative design components of his study. Milgram and his associates planned a series of laboratory experiments in which participants were deceived into believing that they were part of a study of the effects of punishment on learning, and were then told to give increasingly severe electrical shocks to a supposed “subject” who was actually an accomplice of the researchers, and

MIXED METHODS DESIGN IN HISTORICAL PERSPECTIVE

who feigned pain and eventual refusal to cooperate. Chapters 2–4 and 6 provide detailed descriptions of the experimental protocols, including graphs and tables of the results. However, these are interspersed with photographs of the experimental set-up and procedures, and transcripts of participants’ reactions and their dialogue with the experimenters. Chapters 5 and 7, in contrast, present postexperimental interviews with participants. Milgram stated that From each person in the experiment we derive one essential fact: whether he has obeyed or disobeyed. But it is foolish to see the subject only in this way. For he brings to the laboratory a full range of emotions, attitudes, and individual styles.… We need to focus on the individuals who took part in the study not only because this provides a personal dimension to the experiment but also because the quality of each person’s experience gives us clues to the nature of the process of obedience. (1974, p. 44)

This quote provides an early statement of the major complementary strengths of qualitative and quantitative methods: that experimental and quantitative methods are particularly good at showing that a particular intervention or variable resulted in a particular outcome, but qualitative methods are necessary for understanding the processes by which this relationship occurred. (I discuss this in more detail below.) And in addressing potential validity threats to the study’s conclusions, Milgram used both the quantitative results from the experimental manipulations and qualitative data from the observations to rule out these threats. The integration of qualitative and quantitative approaches in design was also present in other fields that have received almost no recognition in the “mixed methods” literature. In anthropology, such integration has been continuously present for many years (Pelto, 2015; Weisner, 2012). Malinowski (1922), in a work that substantially transformed ethnographic field research, emphasized the use of both methods (p. 24), and this has been repeatedly endorsed in methods texts (e.g., Bernard, 1988; Heath & Street, 2008; Herskovits, 1952; see Maxwell, 2016, for a more detailed discussion). Much of this early work (involving such sophisticated quantitative techniques as game theory and Guttman scaling) is discussed in Johnson (1978), in what may be the first textbook on combining quantitative and ethnographic methods in research design. A later development took place in archeology, by proponents of what was initially called

41

the “new archeology’’, later termed “processual archeology” (Binford, 1983), which promoted the systematic testing of theories using statistical methods. Some proponents of this approach also advocated the incorporation in archeology of ethnographic fieldwork with existing communities, along with quantitative methods, to better understand the processes that created the sorts of remains that archeologists study (e.g., Binford, 1978; see Maxwell, 2016, for a more detailed discussion). Similarly, in linguistics, researchers have been integrating qualitative and quantitative methods since the 1960s, but until recently there has been little explicit discussion in published empirical or methodological works of how these can be integrated. Schilling (2013) identified Penelope Eckert’s Linguistic Variation as Social Practice (2000) as “a model study exemplifying the synergistic union of quantitative variationist and qualitative ethnographic methods” (p. 9). Another, unfortunately unrecognized, example is a study by Zentella (1990), one of the most intensive and innovative examples that I’ve seen of combining ethnographic fieldwork with sophisticated quantitative data analysis; see Maxwell (2018) for a detailed discussion of this study. Although (with the partial exception of anthropology) this period exhibits much less explicit discussion of combining qualitative and quantitative approaches, the actual use of both methods in a study was widespread.

THE “PARADIGM WARS” AND THE EXPLICIT EMERGENCE OF “MIXED METHODS” In some fields, however, the 1970s and 1980s saw a sharp polarization of relationships between quantitative and qualitative researchers. Although at least one important work on combining qualitative and quantitative methods appeared during this polarization (Cook & Reichardt, 1979), the increasing sophistication of quantitative methods and accompanying dismissal of qualitative methods (particularly in programme evaluation), led some influential qualitative researchers (particularly Egon Guba and Yvonna Lincoln) to argue that the two approaches were incompatible, because they were based on different “paradigms”—quantitative research on positivism and qualitative on constructivism. In response, prominent advocates for what came to be called “mixed methods” argued that this

42

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

approach required a third paradigm—philosophical pragmatism—which justified combining the two approaches (Biesta, 2010; Johnson & Gray, 2010; Maxcy, 2003). Other philosophical stances were also proposed, including a transformative/ emancipatory paradigm (Mertens, 2018, p.13), but there has been a prevalent view within this community that some paradigm is necessary for mixed methods research. During the formation of a self-identified “mixed methods” community, a typological conception of design also became prevalent, and is currently dominant, as described above. Creswell and Creswell (2020) defined “research designs” as “types of inquiry … that provide specific direction for procedures in a research study” (p. 271). For a description of the evolution of typologies, see also Chapter 2 (this volume). However, alternative conceptions of paradigms and design were also proposed within this community. A particularly strong statement is by Jennifer Greene (2007): [T]he discussion in this book offers a counterpoint to two trends in the broader contemporary mixed methods conversation. In one trend, mixed methods inquiry is importantly defined by its design alternatives, which comprise various methods (usually labeled qualitative and quantitative) arranged in various sequences and priorities . . . Muted by the emphasis on design typologies are the possible contributions to better understanding that could come from mixes of differences in philosophy, substantive theory, and disciplinary thinking, alongside mixes of difference in personal experience, education, values, and beliefs . . . The second trend popular in current mixed methods discourse features the advancement of an “alternative” philosophical paradigm for mixed methods social inquiry; that is, alternative to extant “quantitative” and “qualitative” paradigms . . . It is understandably tempting to identify an “alternative” philosophical paradigm that somehow dissolves or resolves these longstanding debates . . . But I reject this as the only viable response to this challenge, embracing instead several other stances on mixing paradigms while mixing methods. (pp. 15–16)

The design typologies and paradigm debates that have characterized the self-identified “mixed methods” community are largely absent in mixed methods research outside this community—for example, in political science, linguistics, designbased research in education and single-subject/

single-case research in psychology. Each of these fields, as well as the self-identified “mixed methods” community, is substantially “siloed”, having almost no significant interchange with other communities that are integrating qualitative and quantitative methods (Maxwell, 2018).

IMPLICATIONS FOR DESIGNING MIXED METHODS RESEARCH The first implication that I draw from this historical review is that the assumption that a “paradigm” is fundamental to research design, and to mixed methods design in particular, is misplaced. None of the earlier work discussed above, or the more recent developments outside the “mixed method community,” invoked paradigms or paradigm differences. The term “paradigm” was introduced into common use by Thomas Kuhn in his The Structure of Scientific Revolutions (1962), but Kuhn never used the term to mean a philosophical stance that served as a foundation for research. This use was initiated by qualitative researchers Yvonna Lincoln and Egon Guba (1985), who defined what they first called the naturalistic paradigm as the appropriate stance for qualitative research, in opposition to the positivistic paradigm that they associated with quantitative research; Guba and Lincoln (1989) later termed this the “constructivist paradigm”. Many researchers in the mixed methods community accepted this conception of paradigms, but believed that mixed methods research required a different paradigm from qualitative and quantitative research, as described above. However, other mixed methods researchers besides Greene challenged the view that mixed methods needed a single paradigm; see Maxwell (2011) for some of these sources and arguments. The fact that earlier researchers, as well as contemporary researchers outside this community who were integrating qualitative and quantitative methods, saw no need for a single paradigm to serve as a foundation for their work suggests that this may be an unnecessary, and possibly misleading, imposition on mixed methods design. In my view, mixed methods research can be conducted using a variety of philosophical stances, each of which provides insights, but also has limitations (Maxwell, 2011). In particular, I have argued that critical realism (Maxwell, 2012, 2017; Pawson, 2013; Pawson & Tilley, 1997), a stance that has had a significant influence on research methods, particularly in evaluation research, provides a useful perspective for

MIXED METHODS DESIGN IN HISTORICAL PERSPECTIVE

addressing causal processes; this perspective has received little attention in the mixed methods literature. Such philosophical stances are an important part of the researcher’s conceptual framework, and thus of the research design, and have significant consequences for the rest of the design. However, these stances are often implicit and their implications for the rest of the research design need to be made explicit and clearly understood. The second implication of this review is that the central role of design typologies in discussions of design in the self-identified mixed methods community (e.g., Creswell & Plano-Clark, 2018; Morse, 2015; Nastasi et  al., 2010; Tashakkori et  al., 2021, Chapter 5) is problematic. The explicit strategies for integrating qualitative and quantitative methods and data that have recently been developed within the mixed methods community have, to a significant extent, been based on these design typologies. However, the use of both qualitative and quantitative data in developing, as well as testing and supporting, the conclusions of a study, as found in both earlier and contemporary research (Maxwell, 2018) often doesn’t fit the “types” of designs in the mixed methods literature. These types typically either prioritize one method as “dominant” and the other as “subordinate” (e.g., Morse, 2015, pp. 207–208), and/ or have the two approaches used in sequence, e.g., Creswell’s “exploratory” and “explanatory” designs (Creswell & Plano Clark, 2018). This tends to ignore the interactive use of both methods throughout a study, which can contribute substantially to the integration of the two perspectives and lead to deeper understanding of the phenomena studied. Such design types may be helpful in presenting the design to an outside audience. However, existing typologies neglect many key aspects of design that are represented in my interactive model— in particular, the goals, conceptual framework, and validity and generalizability strategies. I’ve addressed this issue in several papers (Maxwell, 2018, 2019; Maxwell et  al., 2015; Maxwell & Loomis, 2003), criticizing typological approaches to design and presenting an alternative conception of design. My argument is that insofar as such typologies are useful at all, they are best seen, not as a “starting point” for research, but as the result of a detailed consideration of your design components and how these interact to shape your research design. A great deal of mixed methods research has been conducted without using the typologies of research designs that have been prominent within the mixed methods community. This not only characterizes the earlier history of mixed methods research; it is also true of much contemporary

43

research. The collection of mixed methods studies edited by Weisner (2005) contains 12 chapters describing mixed methods studies of children and families, many of them programme evaluations, a field that has been closely involved in the explicit development of mixed methods since the important volume by Cook and Reichardt (1979). None of these studies made use of these typologies; more broadly, there are few references to the mixed methods literature in general; the only exception is to Jennifer Greene’s work, which is largely nontypological, and is cited multiple times. Similarly, the studies collected by Hay (2016), in a volume that is subtitled “Integrating mixed methods for more effective social science research”, make no use of these typologies and have almost no citations of authors prominent in the “mixed methods” community. The third implication that I draw from this history is that the relative strengths and limitations of qualitative and quantitative approaches are critical to successful integration, but these have been understood differently in different fields (Maxwell, 2018, p. 320), and in my view the mixed methods literature hasn’t clearly conceptualized these. In particular, quantitative methods are good at showing that a specific intervention or variable caused a given result, and in extending this finding to a randomly sampled population (though with caveats; see Maxwell, 2017). Qualitative methods and perspectives, in contrast, are especially valuable for discovering how it did so—the processes (including participants’ beliefs and mental processes) that led to this result, and the contextual factors that influenced this outcome. Shadish et  al., in what is widely regarded as the pre-eminent work on experimental and quasiexperimental designs, gave a surprisingly clear statement of these differences: [T]he unique strength of experimentation is in describing the consequences attributable to deliberately varying a treatment. We call this causal description. In contrast, experiments do less well in clarifying the mechanisms through which and the conditions under which that causal relationship holds—what we call causal explanation. (2002, p. 9; emphasis in original)

And they stated that “qualitative methods provide an important avenue for discovering and exploring causal explanations” (2002, p. 389). I’ve provided elsewhere (Maxwell, 2020) a more detailed argument for this view of the different strengths of qualitative and quantitative approaches. Bill James’s essay “Jeter vs. Everett” (2006) is a good example (at least for baseball fans) of these differences. Quantitative measures

44

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

strongly suggested that Derek Jeter was not a particularly good defensive short-stop, but video comparison of the play of Jeter and Adam Everett, a shortstop at the opposite end of the quantitative range, not only countered the argument that the quantitative results were simply in error in some way, but also provided an understanding of why Jeter’s defensive play was less effective than Everett’s. In addition, qualitative methods are essential for assessing the generalizability of a study’s conclusions to other populations or settings,2 a task for which quantitative methods provide few useful tools (Cartwright, 2015; Cartwright & Hardie, 2012; see Maxwell, 2020, for a more detailed discussion). Such generalization, for which qualitative researchers typically use the term “transfer” (Donmoyer, 2008; Guba, 1981; Lincoln & Guba, 1985), depends fundamentally on understanding the processes (including mental processes) operating in the original setting, and how these would be influenced by the new context of a different setting—both major strengths of qualitative approaches. Finally, I worry that the emphasis on paradigms and design typologies, and the lack of attention to the wider range of approaches to combining methods in the social sciences, may lead to the marginalization of the self-identified “mixed methods community” in the growing recognition and development of combining methods in the wider research community. For these reasons, I believe that greater attention to the much broader range and diversity of mixed method approaches, and to the more inclusive concept of “design” presented here, would be beneficial to the development of the field of mixed methods research.

WHAT TO READ NEXT Maxwell, J. A., & Loomis, D. (2003). Mixed methods design: An alternative approach. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research, pp. 241–271. Sage.

This chapter presents in more detail my integrative design model. Each of the five components is described in depth, with examples. Maxwell, J. A. (2011). Paradigms or toolkits? Philosophical and methodological positions as heuristics for mixed methods research. Mid-Western Educational Researcher 24(2), 27–30.

This paper challenges the view of mixed methods research as based on a single philosophical or methodological position or “paradigm”. It argues for seeing such positions as conceptual tools that can be used in combination to provide greater insights. Maxwell, J. A., Chmiel, M., & Rogers, S. (2015). Designing integration in mixed method and multimethod research. In S. Hesse-Biber and B. Johnson (Eds.), Oxford handbook of multimethod and mixed methods research inquiry (pp. 223–239). Oxford University Press.

This chapter presents a strategy for integrating qualitative and quantitative approaches and results. A wide range of examples, from the natural as well as the social sciences, provide lessons on how this can be done, and the advantages this strategy creates.

Notes 1  For a more extensive discussion of these and other studies from this period, see Maxwell (2016, 2018); Maxwell et  al., (2015); and Maxwell & Loomis (2003); the last-cited work contains “design maps” of several of these studies. 2  I have labelled this type of generalization external generalization. This is distinct from internal generalization, the generalization of findings from individuals directly interviewed, or settings observed, to the target population or larger settings from which these were drawn (Maxwell, 1992, 2020; cf. Eisenhart, 2009). External generalization, or transfer, is primarily the responsibility of the potential user of the original findings in a new context, although the original author can facilitate this transfer by providing detailed information about the processes that led to the reported results, and how these were influenced by the original context.

REFERENCES Aristotle’s biology. Stanford Encyclopedia of Philosophy. Accessed at: https://plato.stanford.edu/entries/ aristotle-biology/ Becker, H. S., Geer, B., Hughes, E. C., & Strauss, A. L. (1961). Boys in white: Student culture in medical school. Transaction Books. Bernard, H. R. (1988). Research methods in cultural anthropology. Sage.

MIXED METHODS DESIGN IN HISTORICAL PERSPECTIVE

Biesta, G. (2010). Pragmatism and the philosophical foundations of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social & behavioral research (2nd ed., pp. 95–117). Sage. Binford, L. (1978). Nunamiut ethnoarchaeology. Academic Press. Binford, L. (1983). In pursuit of the past: Decoding the archeological record. Thames & Hudson. Blau, P. (1963). The dynamics of bureaucracy (rev. ed.). University of Chicago Press. Cartwright, N. (2015). Where’s the rigor when you need it? Foundations and Trends® in Accounting 10(2–4), 106–124. http://dx.doi.org/10.1561/ 1400000045 Cartwright, N., & Hardie, J. (2012). Evidence-based policy: A practical guide to doing it better. Oxford University Press. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues for field settings. Houghton Mifflin. Cook, T. D., & Reichardt, C. S. (1979). Qualitative and quantitative methods in evaluation research. Sage. Creswell, J. W., & Creswell, J. D. (2020). Research design: Quantitative, qualitative, and mixed methods approaches (5th ed.). Sage. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Creswell, J. W., & Poth, C. (2017). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). Sage. Darwin, C. (1872). The expression of the emotions in man and animals. John Murray. Denzin, N. K, & Lincoln, Y. S. (2000). Handbook of qualitative research. Sage. de Waal, F. (2016). Are we smart enough to know how smart animals are? W. W. Dutton & Co. Donmoyer, R. (2008). Generalizability. In L. Given (Ed.), The SAGE encyclopedia of qualitative research methods (pp. 371–372). Sage. DuBois, W. E. B. (1899). The Philadelphia Negro: A social study. University of Pennsylvania Press. Eckert, P. (2000). Linguistic variation as social practice. Blackwell. Eisenhart, M. (2009). Generalization from qualitative inquiry. In K. Ercikan & W.-M. Roth (Eds.), Generalizing from educational research (pp. 51–66). Routledge. Festinger, L., Riecken, H. W., & Schachter, S. (1956). When prophecy fails. University of Minnesota Press. Freedman, D. A. (1991/2010). Statistical models and shoe leather. In D. Collier, J. S. Sekhon, & P. B. Stark (Eds.), Statistical models and causal inference (pp. 45–62). Cambridge University Press. (Reprinted

45

from Freedman, D. A., Sociological Methodology, 21, pp. 291–313.) Freedman, D. A. (2008). On types of scientific inquiry: The role of qualitative reasoning. In J. M. Box-Steffensmeier, H. E. Brady, & D. Collier (Eds.), Oxford handbook of political methodology (pp. 300–318). Oxford University Press. Greene, J. C. (2007). Mixed methods in social inquiry. Jossey-Bass. Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. Educational Communication and Technology, 29, 79–92. Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. SAGE. Hagood, M. J. (1939/1996). Mothers of the South: Portraiture of the White tenant farm woman. University of Virginia Press. Hay, M. C. (Ed.) (2016). Methods that matter: Integrating mixed methods for more effective social science research. University of Chicago Press. Heath, S. B., & Street, B. V. (with Mills, M.). (2008). Ethnography: Approaches to language and literacy research. Teachers College Press. Heath, T. L. (1932/1991). Greek astronomy. Dover Publications. Herskovits, M. J. (1952). Economic anthropology: The economic life of primitive peoples. W. W. Norton. Jabr, F. (2010). The evolution of emotion: Charles Darwin’s little-known psychology experiment. https://blogs.scientificamerican.com/observations/ the-evolution-of-emotion-charles-darwins-littleknown-psychology-experiment/ Jahoda, M., Lazarsfeld, P. F., & Zeisel, H. (1971). Marienthal: The sociography of an unemployed community. Aldine Atherton. (Original work published in German in 1933.) James, B. (2006). Jeter vs. Everett. Retrieved from: http://fieldingbible.com/jeter.asp Johnson, A. W. (1978). Quantification in cultural anthropology: An introduction to research design. Stanford University Press. Johnson, R. B., & Gray, R. (2010). A history of philosophical and theoretical issues for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social and behavioral research (2nd ed., pp. 69–94). Sage. Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press. Lazarsfeld, P. F. (1971). Forty years later. In Jahoda, M., Lazarsfeld, P. F., & Zeisel, H., Marienthal: The sociography of an unemployed community, pp. xxxi–xl. Aldine Atherton. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Sage. Lynd, R. S., & Lynd, H. M. (1929) Middletown: A study in American culture. Harcourt, Brace.

46

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Malinowski, B. (1922). Argonauts of the western Pacific. E. P. Dutton. Maxcy, S. J (2003). Pragmatic threads in mixed methods research in the social sciences: the search for multiple modes of inquiry and the end of the philosophy of formalism. In A. Tashakkori & C. Teddlie, SAGE handbook of mixed methods in social & behavioral research (pp. 51–89). Sage. Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, 62(3), 279–301. http://dx.doi.org/10.17763/ haer.62.3.8323320856251826 Maxwell, J. A. (2010). Using numbers in qualitative research. Qualitative Inquiry, 16(6), 475–482. https://doi.org/10.1177%2F1077800410364740 Maxwell, J. A. (2011). Paradigms or toolkits? Philosophical and methodological positions as heuristics for mixed methods research. Mid-Western Educational Researcher 24(2), 27–30. Maxwell, J. A. (2012). A realist approach for qualitative research. Sage. Maxwell, J. A. (2013). Qualitative research design: An interactive approach (3rd ed.). Sage. Maxwell, J. A. (2016). Expanding the history and range of mixed methods research. Journal of Mixed Methods Research, 10 (1), 12–27. https:// doi.org/10.1177/1558689815571132 Maxwell, J. A. (2017). The validity and reliability of research: A realist perspective. In D. Wyse, N. Selwyn, E. Smith, and L. E. Suter (Eds.), The BERA/SAGE Handbook of Educational Research, pp. 116–140. Sage. Maxwell, J. A. (2018). The ‘silo problem’ in mixed methods research. International Journal of Multiple Research Approaches, 10(1), 317–327. https:// doi.org/10.29034/ijmra.v10n1a20 Maxwell, J. A. (2019). Distinguishing between quantitative and qualitative research: A response to Morgan. Journal of Mixed Methods Research, 13(2), 132–137. https://doi.org/10.1177/155868 9819828255 Maxwell, J. A. (2020). The value of qualitative inquiry for public policy. Qualitative Inquiry 26(2), 177– 186. https://doi.org/10.1177/1077800419857093 Maxwell, J. A., Chmiel, M., and Rogers, S. (2015). Designing integration in mixed method and multimethod research. In S. Hesse-Biber and B. Johnson (Eds.), Oxford handbook of multimethod and mixed methods research Inquiry (pp. 223–239). Oxford University Press. Maxwell, J. A., & Loomis, D. (2003). Mixed methods design: An alternative approach. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 241–271). Sage. Mertens, D. N. (2018). Mixed methods design in evaluation. Sage.

Milgram, S. (1974). Obedience to authority: An experimental view. Harper & Row. Molina-Azorin, J. F., & Fetters, M. D. (2022). Books on mixed methods research: A window on the growth in number and diversity. Journal of Mixed Methods Research, 16(1), 8–16. https://doi. org/10.1177/15586898211068208 Morse, J. M. (2015). Issues in qualitatively-driven mixed method designs: Walking through a mixedmethod project. In S. Hesse-Biber & R. B. Johnson, The Oxford handbook of multimethod and mixed methods research inquiry (pp. 206–222). Oxford University Press. Nastasi, B. K., Hitchcock, J. H., & Brown, L. M. (2010). An inclusive framework for conceptualizing mixed methods design typologies: Moving toward fully integrated research models. In A. Tashakkori & C. Teddlie, SAGE handbook of mixed methods in social & behavioral research (2nd ed., pp. 305–338). Sage. Pawson, R. (2013). The science of evaluation: A realist manifesto. Sage. Pawson, R., & Tilley, N. (1997). Realistic evaluation. Sage. Pelto, P. J. (2015). What is so new about mixed methods? Qualitative Health Research, 25, 734–745. https://doi.org/10.1177/1049732315573209 Platt, J. (1996). A history of social research methods in America, 1920–1960. Cambridge University Press. Poth, C. N. (2018). Innovation in mixed methods research. Sage. Roethlisberger, F. J., & Dickson, W. J. (1939). Management and the worker. Harvard University Press. Schilling, N. (2013). Sociolinguistic fieldwork: Key topics in sociolinguistics. Cambridge University Press. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin. Tashakkori, A. M., Johnson, R. B., & Teddlie, C. B. (2021). Foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences (2nd ed.). Sage. Teddlie, C. B., & Tashakkori, A. M. (2006). A general typology of research design featuring mixed methods. Research in the Schools, 13(1), 12–28. Warner, W. L., & Lunt, P. S. (1941). The social life of a modern community. Greenwood Press. Webster’s Ninth New Collegiate Dictionary (1984). G. & C. Merriam Company. Weisner, T. (Ed.) (2005). Discovering successful pathways in children’s development: Mixed methods in the study of childhood and family life. University of Chicago Press.

MIXED METHODS DESIGN IN HISTORICAL PERSPECTIVE

Weisner, T. (2012). Mixed methods should be a valued practice in anthropology. Anthropology News, 53(5), 3–4. Wikipedia (n.d.). Aristotle’s biology. Accessed at: https://en.wikipedia.org/wiki/Aristotle’s_biology Zeisel, H. (1933/1971). Afterword: Toward a history of sociography. In M. Jahoda et  al., Marienthal:

47

The sociography of an unemployed community (pp. 99–125). Aldine Atherton. Zentella, A. C. (1990). Integrating qualitative and quantitative methods in the study of bilingual code switching. Annals of the New York Academy of Sciences, 583, 75–92. https://doi.org/10.1111/ j.1749-6632.1990.tb12186.x

4 Mixed Methods Designs to Further Social, Economic and Environmental Justice Donna M. Mertens

Gross economic disparities, increasing environmental disasters and violations of human rights bring to visibility systems that need to be transformed in order to provide conditions for an improved quality of life for all. Mixed methods researchers are in a strong position to address these issues because the integration of multiple methods generates new questions, engages diverse groups and supports enhanced understandings of complexity in addressing these wicked problems. However, the conceptualization and implementation of mixed methods to these ends is not without tensions because of the multiplicity of paradigmatic lenses that characterize the mixed methods community (Mertens, 2018a). It is possible for mixed methods researchers in all the major paradigmatic schools to address the issues of justice. However, the transformative mixed methods lens provides an approach to mixed methods that is based on the assumption that the role of the researcher is to support increased social, economic, and environmental justice. The research community does not commonly focus on the intersection of these three types of justice, meaning this is fertile ground for advancements and innovation in mixed methods designs. I make the argument that mixed methods researchers who apply the transformative lens to their work can design studies that are consciously

focused on addressing inequities in a culturally responsive way. Mixed methods researchers who commit to being culturally responsive integrate culture, history, and diversity in terms of race/ethnicity, linguistic variations, disability, gender and other characteristics that create cohesion in social groups (Hood et al., 2015). This takes the researcher beyond a step-by-step application of mixed methods designs to territory that requires rethinking the role of the researcher. Researchers can make a choice to be complicit in sustaining an oppressive status quo or to support the transformations that are needed to increase social, economic and environmental justice. Researchers who accept responsibility to work for increased justice need to shift their role from a traditional creator of knowledge to one that integrates aspects of being a social change agent (Dhaliwal, et  al., 2020; McBride et al., 2020). The importance of addressing the intersection of social, economic, and environmental justice provides a strong rationale for applying a transformative lens to mixed methods research. These three types of justice represent values that are not easily defined and, consequently, researchers who choose to contribute to transformative change face significant challenges. Justice is commonly seen as a value associated with fairness and equity. When social justice is considered, it is discussed

Mixed Methods Designs to Further Social, Economic and Environmental Justice 49

in terms of promoting a more inclusive, progressive and equal society free from discrimination and supportive of fundamental human rights, such as access to healthcare, education, and safety and security (Levin, 2020). Economic justice is inclusive of access to opportunities for work that is safe and fairly compensated, ownership of property and assets, liberty from want, and power of decisionmaking concerning employment. According to Robert Bullard, the acknowledged founder of the Environmental Justice Movement, environmental justice focuses on equal access to clean water, air and earth, and is inclusive of such areas as housing, transportation, energy, food security, climate and sustainability (Mohai et al., 2020). The United Nations Evaluation Group recognized the importance of the intersection of social, economic and environmental justice. They reported that “Vulnerable people are seen as suffering more than others from environmental degradation, exploitation, climate change and natural or man-made disasters and related risks” (Spilsbury, 2020, p. 16). Rising temperatures are associated with increased numbers of and severity of natural disasters with the burden falling most heavily on poor countries in terms of deaths and economic losses (UN Office for Disaster Risk Reduction, 2020). In the United States, the negative impact of the climate crisis disproportionally affects people of color and Indigenous populations (Deitz & Meehan, 2019). When economic goals are prioritized without consideration for environmental and social impacts, the result can leave people worse off than they were before (Widianingsih & Mertens, 2019). The transformative lens applied to mixed methods provides a pathway to address this important intersection because it involves multiple constituencies and uses multiple methods for transformative purposes. Another important aspect of transformative mixed methods is planning for increased impact and sustainability. Transformative mixed methods designs have incorporated these action-oriented aspects in the form of specifically including members of marginalized and vulnerable communities in culturally responsive ways. Also, researchers build coalitions that are consciously inclusive and address power inequities as strategies to increase impact and sustainability. These strategies are illustrated in the examples presented later in the chapter. At the same time, it is worth noting that studies that use a transformative approach to address the intersection of social, economic, and environmental justice are rare; this clearly is an area for growth in the research community. The transformative approach to research emerged in response to concerns expressed by members of vulnerable and marginalized populations that

research was not only inaccurately representing their communities, but was in many cases, leaving them no better or worse off (Mertens, 2018a, 2020a; Mertens & Wilson, 2019). For example, Indigenous people hold that “the ‘bad name’ that research has within Indigenous communities is not about the notion of research itself; rather, it is about how that research has been practised, by whom and for what purpose that has created ill-feeling” (Cram et al., 2013, p. 11). Use of a transformative lens is a conscious strategy to work toward positive transformations, rather than simply describing a phenomenon or “giving voice” to stakeholders.

TRANSFORMATIVE ASSUMPTIONS AND MIXED METHODS The transformative lens is defined by assumptions about ethics and values (axiology), the nature of reality (ontology), the nature of knowledge and relationships within the research realm (epistemology), and the nature of systematic enquiry (methodology) (Mertens, 2018a, 2020a). In this section, the reader will explore the methodological assumptions that are derived from the transformative axiological, ontological and epistemological assumptions.

Transformative Axiological Assumption The transformative axiological assumption is defined as the view of ethics and values that is necessary for research to contribute to increased social, economic and environmental justice. “The salient values that constitute this assumption include: cultural respect; explicitly addressing inequities; inclusion of reciprocity (i.e., giving back to the community); an action-orientation towards transformation; recognition of community resilience and the interconnectedness of all things (living and nonliving); and building relationships. (Widianingsih & Mertens, 2019, p. 28)

The implications for methodology for mixed methods researchers rests in the use of mixed methods to develop strategies that are inclusive of all voices in culturally responsive ways. Researchers cannot assume that they understand the problem and the correct solutions; they need to develop strategies based on an understanding of the community that can contribute to positive change. An example

50

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

of the application of the transformative axiological assumption is seen in the approach to court action following the arrest of persons with mental health and substance abuse problems (Mertens, 2018b). A lack of cultural respect results in high rates of incarceration and recidivism. When the transformative ethical lens is used, the court recognizes community strengths and cultural differences that provide guidance for supportive services to keep people out of jail and on the road to recovery.

Transformative Ontological Assumption The transformative ontological assumption follows from the transformative axiological assumption in recognizing that there are different versions of reality that emanate from different social positionalities. Some versions of reality sustain an oppressive status quo; others provide a pathway to increased social, economic, and environmental justice. For example, the building of a textile plant in Bandung West Java to address economic issues was a version of reality that did not take into consideration its negative environmental and social impacts (Widianingsih & Mertens, 2019). The result was the creation of the most polluted river in Indonesia because of the chemicals from the plant; this was the river where the people bathed and from which they took their drinking water. The negative impact was witnessed in the increase in health problems for the people who depended on that river. Transformative mixed methods researchers can serve communities by collecting quantitative and qualitative data on the anticipated consequences of accepting an oppressive version of reality over one that contributes to increased justice.

Transformative Epistemological Assumption The epistemological assumption is concerned with the nature of knowledge and the relationship between researchers and community members. The transformative epistemological assumption calls for honoring the experiential knowledge of persons with lived experience and for explicitly addressing power inequities in order to provide a safe space for inclusion of all stakeholders, including members of vulnerable and marginalized communities (Mertens, 2020a). The transformative epistemological assumption leads to asking questions about whose knowledge counts and about the

historical factors that influence value accorded to such knowledge. The methodological implications of the transformative epistemological assumption include collection of qualitative and quantitative data on the historical, cultural and contextual issues that inform the building of relationships with an understanding of who needs to be included and how to include them respectfully. Another methodological implication is that researchers need to consciously structure culturally responsive relationships that allow for full inclusion. Strategies for working together need to be developed. In some cases, researchers can support the development of coalitions that can serve to increase understandings of problems, inform development of solutions, increase impact and contribute to sustainability (Wolfe et  al., 2020). Widianingsih and Mertens (2019) provide an example of building a coalition of farmers, their wives and the young people in the farming community to bring their concerns to the government so that services could be more responsive to their needs. This coalition informed the process of data collection, interpretation and use of results, and provided for sustainability in their efforts for transformation.

Transformative Methodological Assumption The transformative methodological assumption embodies the methodological implications of the other three assumptions. A transformative mixed methods design needs to incorporate culturally responsive strategies that support positive change to increase social, economic, and environmental justice. The use of a transformative lens combined with the mixed methods (both quantitative and qualitative integrated into the design) provides opportunities to be responsive to multiple stakeholder groups and to capture the complexity of the processes needed for transformative change. The design also needs to incorporate the building of relationships and use of the findings throughout the study to critically examine and inform practices and policies. (Widianingsih & Mertens, 2019, p. 28)

Figure 4.1 displays a multi-stage transformative mixed methods design. The assumption is that a cyclical design allows for information collected at each stage to inform actions in subsequent stages. For example, development of a coalition of grassroots community members, policy makers and service providers could contribute ideas to data

Mixed Methods Designs to Further Social, Economic and Environmental Justice 51

Phase 1: Rela�onship building

Phase 2: Contextual analysis

Phase 3: Pilot tes�ng

economic or social issues, with environmental factors not addressed or addressed as an afterthought. Therefore, the examples used in this section will highlight strategies that researchers have used or could use to increase the impact of their work to increase justice, such as developing culturally responsive inclusive relationships, coalition building, adopting strategies from social activists and social change agents, addressing power issues and planning for sustainability.

Culturally Responsive Relationships Phase 4: Implementa�on

Phase 5: Determine effec�veness

Phase 6: Use for transforma�ve purposes

Figure 4.1  Multi-stage transformative mixed methods design Source: Adapted from Mertens (2018a).

that would be useful in the contextual analysis stage. The arrows between the stages are doubleheaded to indicate that each stage can be revisited throughout the process of the study when new information comes to light. In the following section, examples of transformative mixed methods studies are used to illustrate aspects of this approach that have particular relevance for increasing research impact in service to social, economic, and environmental justice. As the reader will see, many challenges exist in conducting transformative mixed methods research and there is a dearth of examples in which the nexus of social, economic and environmental justice is brought together.

STRATEGIES TO INCREASE IMPACT FOR SOCIAL, ECONOMIC AND ENVIRONMENTAL JUSTICE Researchers rarely address the nexus of social, economic and environmental impact (Dobinger, 2021). More often than not, the focus is on

The development of culturally responsive relationships requires researchers to invest time in identifying who needs to be included and to develop strategies that provide a safe space that acknowledges the knowledge and experience brought by community members. Cultural responsiveness also entails achieving an understanding of the context in terms of history, legislation and other relevant factors. For a discussion of culturally responsive mixed methods applied to an evaluation context, please see also Chapter 27 (this volume). This strategy of cultural responsiveness is illustrated by a study of a court diversion program for people who were arrested for substance abuse and who have a mental illness (Mertens, 2018b). A contextual analysis looked at the history of responses to drug addiction as part of this transformative mixed methods study. Document reviews revealed that in many countries, policies related to drug addiction are connected with punishment for criminal behavior. In the 1980s and 1990s, the United States addressed substance abuse with policies known as the “war on drugs” (Washington Post, 2015). These policies included harsh mandatory sentencing that resulted in mass incarceration for many nonviolent offenders (Mertens, 2018b). The heaviest burden fell on communities of color who were arrested more often than white people who engaged in the same behaviors. Prisons in the United States are occupied disproportionately by people of color based at least partly on the “war on drugs” policy being inequitably applied. Information about the policies, their historical use, and the extant data on sentencing and characteristics of prison populations illustrate the type of contextual analysis that would inform a transformative approach for researchers who want to contribute to a more just system for dealing with drug addiction. The Marion County Mental Health Court in Salem Oregon, with presiding Judge Mary James,

52

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

wanted to develop a more just approach for people who had been arrested on drug charges who also presented with symptoms of mental illness (Mertens, 2018b). Approaching this challenge with a transformative lens, Judge James built relationships with a wide range of stakeholders, as recommended by Slate (2003). These included her staff, clinical caseworkers, administrators linked to public and/or private mental healthcare providers in the community, representatives from both the prosecution and the defense, law enforcement personnel (including jail administrators and probation officials), mental health advocates and consumers of mental health services. This group served as a coalition to support program participants and to sustain the outcomes of the program. Additional contextual analysis revealed legislative and policy support for changing the perspective (version of reality) regarding drug addiction away from a “war on drugs” to one that viewed drug dependency and mental health issues as public health problems (Slate, 2003). The US Congress passed America’s Law Enforcement and Mental Health Act (Public Law 106–515, 2000) that authorized funding for mental health courts and jail diversion programs. Jail diversion programs serve to support people who have drug addiction and co-occurring mental illness by linking them to community-based treatment and support systems (Crisanti, et al., 2014). The Marion County Mental Health Court used this historical and current data as a basis for developing an intervention that was designed to provide a diversion program in the form of an intensive circle of support (Mertens, 2018b). Issues of poverty, homelessness, child-care support, employment, access to healthcare and medications, and healthy food needed to be addressed. They recognized that the same intervention was not appropriate for all offenders who presented in the court with substance abuse and co-occurring mental illness. For example, sexual minorities might be afraid to discuss their sexuality in open court; they might be reticent to report abusive behaviors that they experience because of their sexuality. Hence, quantitative data can be collected about the provision of services, uptake of services, and recidivism rates, but qualitative data is needed in an on-going way in order to know if the participants are receiving the support they need in ways that truly give them a better chance for success. Judge James led a team that collected data about the quality of the experiences for service providers, court personnel, and participants that allowed them to see what needed to be adjusted throughout the program. For example, court personnel and service providers needed additional training on how to be culturally responsive to sexual minorities and

veterans, especially those who suffer from posttraumatic stress disorder (PTSD). The impact of this program was strengthened by informing the intervention through respectful relationships, contextual understanding, and on-going feedback. As Judge James said: “If mental illness is treated in a restorative way and by accepting the responsibility to take a more restorative justice approach, we are going to have people who come out of a criminal justice system with a better feel for how to move forward as part of the community” (Judge Mary James, Interview Notes, 2017, in Mertens, 2018b, p. 9). A second example of developing culturally responsive relationships as part of a transformative mixed methods study is provided by Love et al. (2019) who conducted a qualitatively dominant transformative mixed methods study with drug users in the UK criminal justice system that focused on respectful engagement and policy implications. Love and colleagues traced the history of policies related to drug use and criminalization in the UK to gain historical perspective. A sequential mixed methods design was used with a quantitative study being conducted first that collected data on program aftercare effectiveness in terms of drug misuse and offending behaviors. The government deemed the findings of this study too sensitive to release until seven years later. The study revealed that participants had reduced their use of illegal drugs, but they consumed large amounts of alcohol and other drugs. Policy makers then had to respond to questions about the program’s focus on only Class A drugs. Subsequently, the authors conducted a qualitative study using face-to-face interviews and focus groups led by the rationale that policy needed to be based on a fuller understanding of drug use, relapse and recovery. The researchers engaged in discussions with the drug intervention program staff to build trust and to identify safe procedures for engaging with participants. They used several strategies to build safe relationships with participants such as asking through focus groups about the best way to approach sensitive topics in interviews. During interviews and focus groups, participants were assured that they did not have to answer questions and they were offered counselling afterward to address potential re-traumatization. Focus groups were structured to avoid including members of rival gangs and to avoid having victims and perpetrators of sexual abuse in the same group. Through attention to these legal and ethical considerations, the researchers were able to develop respectful relationships that resulted in information that policy makers and program staff could use to develop more informed policies to support persons with drug addictions.

Mixed Methods Designs to Further Social, Economic and Environmental Justice 53

Coalition Building As noted in the introductory section of this chapter, researchers who accept responsibility to work for increased justice can benefit from effective strategies that are used by social activists and social change agents (Dhaliwal, et  al., 2020; McBride et al., 2020). Coalitions make up a fundamental part of the work of social activists and change agents. However, they have not been a frequent part of mixed methods studies. However, an increasing number of funding agencies are requiring coalitions as part of the funded programs, for example, Health Resources and Services Administration’s Healthy Start Program to address health disparities in communities of color and the Centers for Disease Control in their cancer control programs (Wolfe et  al., 2020). These coalitions are not always structured to be an integral part of the research team. However, as a community of researchers, we can learn from their experiences to improve the impact of our work. One example of coalition building comes from a poverty reduction and safe working conditions initiative that was not part of a research study. Agricultural workers in Florida worked under harsh conditions, characterized by long hours, low pay, sexual harassment and abuse, lack of sanitation facilities and fear of being fired if they spoke up (Greenwood, 2019). The initiative began with two students of law doing community outreach in the agricultural fields of Florida. What they saw motivated them to seek contact with others who wanted to work to improve conditions. They first began their work when they identified three Haitian peasant organizers who were working in the fields who identified several other Latino workers. This small group organized a meeting in a Catholic church; their connections grew into the Coalition of Immokalee Workers and included the Catholic church, foundations, and students at 300 universities and 50 high schools. Other powerful groups joined the coalition, including the National Council of Churches, the Presbyterian Church, and dozens of synagogues. The coalition used many strategies to bring to light the dangerous conditions for farmworkers and to push for changes (Greenwood, 2019). Some of these included educating the workers about their rights and putting pressure on the companies that bought the tomatoes to buy only from farms that respected the workers’ rights. They used weekly skits to educate the workers about their rights and develop their leadership skills. They set up a radio station that broadcast information to the workers and their advocates. They also designed a website to disseminate videos and daily dispatches. They took more radical actions that are not typically

associated with mixed methods research, but might be the missing link to seeing greater impact of such research. For example, a small group of workers went on a hunger strike; workers went out on a full-scale strike; and they staged a two-week protest that involved a 200-mile march. They also demonstrated outside shareholder meetings for big corporations that buy tomatoes, and students led a national boycott against fast-food restaurants that purchased tomatoes from these farms. While some of these actions may seem to be outside the purview of a mixed methods research study, the results of their actions demonstrate the power of applying these strategies for systemic change. The coalition developed an ethical code of conduct that was not just a piece of paper (Greenwood, 2019). It delineated the standards for fair pay, breaks during the day, access to clean water and sanitation, and mechanisms for reporting abuse. This ethical code became the foundation for the Campaign for Fair Food and the Fair Foods Standards Council. The Council is an oversight group that ensures that growers adhere to the ethical code and that fast food restaurants buy their tomatoes from growers who follow the code. There is a 24-hour hotline for workers to lodge a complaint if the code is violated, and they have a staff of investigators to follow up on the complaints. Lessons to be learned from this example of social activism include the need to build a broad coalition, be inclusive of those who have been traditionally marginalized as well as traditional institutions with power, ensure that effective communication occurs throughout the entire process, implement a strategy for sustainability, and have a mechanism for monitoring continuous improvement. The use of coalitions and social change agent strategies provides mechanisms for addressing power issues and planning for sustainability, necessary elements for transformative change.

Addressing Power Inequities and Planning for Sustainability Power inequities in the research world typically favor those in formal positions of power such as researchers at academic institutions, consultants at private corporations, policy makers, and program developers and funders over members of marginalized and vulnerable communities (McBride et al., 2020). A transformative lens calls researchers to give serious consideration to who needs to be included, how to include the full range of stakeholders in culturally responsive and supportive ways, and to the development of strategies that

54

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

challenge the existing power structure. The use of a transformative lens in the design of mixed methods studies can be included in other research designs such as the communicative methodology, participatory action research, community-based participatory research and culturally responsive research (Chouinard & Cram, 2020). In this section, I explore those aspects of mixed methods designs that explicitly address power inequities and planning for sustainability. A group of researchers in Spain developed and implemented the communicative methodology in order to address critical social issues such as discrimination against the Roma population in education (Flecha, 2014; Sordé Marti & Mertens, 2014) and reduction of gender-based violence in universities (Puigvert et al., 2017). Although the research studies occurred in different contexts, they shared a common methodology with specific strategies to address power differences and sustainability: “Communicative methodology frames research as a form of egalitarian dialogue that involves constructing knowledge on the basis of intersubjectivity and joint reflection to identify the actions that have helped reverse social exclusion” (Flecha, 2014, p. 246). In Flecha’s (2014) report of the use of a transformative mixed methods communicative design, he describes the Roma population as the most important nonmigrant ethnic minority in Europe who are the object of multiple forms of discrimination. The methodology used in the Roma community was a long-term case study with both quantitative surveys and qualitative interviews to identify educational actions that contribute to promoting academic achievement, inclusion, and empowerment in a local school that had a history of failure. The key to addressing power differences was inclusion of members of the Roma community as part of the research team. The university researchers needed to build trust by being present in the community, supporting Roma parents and students who wanted to see their schools improve, revising data-collection strategies based on community input, and sharing findings with community members for discussion of interpretation and recommendations. The impact of the research was demonstrated by increased family involvement in the school, higher academic achievement, and stronger family relationships. Qualitative data documented a fundamental transformation: the existing school that was hostile and ineffective was closed and a new school was established in the same place that was developed based on participatory dialogue with local residents. This new school engages with the local Roma community who hold leadership positions; thus, sustainability was planned for and achieved.

Puigvert et al. (2017) provide another example of the use of a transformative mixed methods communicative design to study resistance to and transformation of programs to address gender-based violence in Spanish universities. They used strategies similar to those reported by Flecha (2014) in that a participatory process was used to engage with diverse stakeholders that included dialogues in which students, faculty, and staff could create results in their own contexts. Puigvert et al.’s contribution to addressing power inequities and planning for sustainability is rooted in challenging the oppressive systems that they encountered when they embarked on their study. The researchers were threatened and false claims about their use of sexual favors to advance their careers were circulated on social media. They overcame this hostility by focusing on key informants and reassuring participants that their responses would be anonymous. Their actual methods of data collection included an initial quantitative questionnaire, followed by in-depth interviews and sessions where the participants could tell their communicative daily life stories. Integration of data from these sources revealed that gender-based violence was pervasive, universities did not have measures to respond to this, and they were resistant to addressing the issue. The findings were used to create spaces of support within universities and to introduce the topic to public debate. The researchers continued dialogue with the diverse stakeholders over several years and shared their results through academic conferences, government agencies and public media. This created pressure on universities to develop equality units and protocols for prevention of and response to sexual harassment. The participants also reported increased solidarity and support for students who experienced genderbased violence. Miller (2020) illustrates how to address power issues in a highly stigmatized marginalized group in Zimbabwe—i.e., gay and bisexual men. These men experience oppression in a society that discriminates against them, but also because Zimbabwe has a law that punishes same-sex sexual behavior with up to one year in prison. The researcher worked with an advocacy group and six identity-based collectives. They signed a Memorandum of Understanding to allow gay and bisexual men to visit two major health centers as “mystery shoppers” to collect data to improve access to and experience with healthcare. The study began with a survey of the gay and bisexual men to understand their experiences of stigma and discrimination. The results were used to develop a training program to reduce the men’s self-stigma as a means to support them to seek care. Power issues were directly addressed by identifying ten

Mixed Methods Designs to Further Social, Economic and Environmental Justice 55

young men to serve as leaders to mobilize community members, link people to care, and monitor the quality of care. An instrument was developed through an interactive workshop with gay and bisexual men that could be used to report on the quality of their healthcare experiences. This checklist was used by “mystery shoppers” to rate their experiences; it was also shared with the advocacy group and the healthcare providers. They also identified ten individuals who worked in the centers who served as “champions” who advocated for internal changes based on the results of the mystery shopper data. The peer mobilizers, center staff, champions and healthcare workers reviewed the data, met to discuss implications and worked to develop corrective action plans. The changes resulted in a fourfold increase in the number of gay and bisexual men who sought healthcare. The facilities made the following changes: The changes put into place included sensitivity training policies for new employees, posting welcoming signage that openly affirms LGBTQI people and their right to be in the setting, offering a weekly clinic held in the Sexual Rights Centre compound, and the crafting of an internal advocacy campaign by champions that Mpilo’s nursing program systematically cover material on affirming and non-stigmatizing care. The effort also led the citywide system of 29 neighborhood public health clinics in Bulawayo to request to join the project. A Memorandum of Understanding with the city public health clinics was established in summer of 2019. (Miller, 2020, p. 13)

These impacts were documented through interviews, observations and document reviews. Thus, the impact of the research was increased by building capacity for this population, forming a collective, developing leaders from the community, identifying champions within the healthcare organizations and involving the full range of stakeholders in the development of instruments and interpretation and use of the findings. Lucero et  al. (2018) demonstrated the use of a community-based participatory research (CBPR) approach to address health disparities for Indigenous Americans. The mixed methods study used an Indigenous-transformative lens that included integration of qualitative and quantitative data at all stages of the study (Chilisa & Mertens, 2020). While Lucero et  al.’s study occurred over a seven-year time frame, I focus here on those aspects that consciously addressed issues of power when working in this community. The study included a comprehensive contextual analysis of political, structural, health, and capacity issues, and histories of collaborations to gain

an understanding of how to structure the study to build trust. They formed partnerships with community and academic CBPR experts, and local and national organizations that represented Indigenous people. A web-based quantitative survey and community focus groups informed the research approach and culturally responsive dynamics of interaction. As the research progressed, the National Congress of American Indians Policy Research Center came to play a leadership role, ensuring that the study was culturally responsive for Indigenous communities. A Scientific Community Advisory Council was formed: “members included university faculty, staff from community-based organizations, advocates, public health workers, and tribal staff and community members” (Lucero et  al., 2018, p. 61). Frequent group consultations resulted in providing meaningful engagement with the Indigenous community members. The outcomes of the study included policy changes, the development of a culturally responsive and sustainable intervention, changed power relations, capacity development for partnering agencies, and reduced health disparities.

SUPPORTING ENVIRONMENTAL, ECONOMIC AND SOCIAL JUSTICE The studies that are presented in this chapter illustrate many strategies that researchers could use to increase the impact of their work to increase environmental, economic and social justice. However, these studies do not address this intersection explicitly. The lack of attention to this space was noted by Dobinger (2021). This is an area in serious need of attention given the relationship between environmental factors and their disproportionately negative effects on vulnerable and marginalized populations (Spilsbury, 2020; UN Office for Disaster Risk Reduction, 2020; Yeampierre & Klein, 2020). For example, health disparities for communities of color may be narrowly focused on provision of health services. However, this ignores the complex realities that contribute to health disparities: racism in the form of policies that limit economic opportunities and force people of color to live in areas that have higher levels of environmental pollutants (Rothstein, 2017). Large corporate buy-outs of small farmers’ land in Indonesia resulted in an influx of poor people into urban areas (Widianingsih & Mertens, 2019). These people found that it was difficult to find work in the cities; some turned to illegal activities and some were trafficked to other countries where they

56

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

faced abuse. One economic solution that was put into place, as mentioned previously, was a textile factory that created jobs but at high environmental and social costs. Addressing these complex problems is not easy, but ignoring them leaves researchers at risk of being complicit in sustaining oppression. Brousselle and McDavid (2020) suggest a pathway for researchers to expand their role and address the complexities presented when considering environmental, economic, and social in justice. They assert that solutions to such problems exist, but the issue now is “in securing commitments for action, commitments that are implemented and sustained over time” (p. 2). They suggest that researchers ask questions about planetary health as a part of framing their studies. When considering the natural environment, researchers should consider how an intervention would affect pollutants in the air, soil, and water; preservation of land and water; and preservation of the lives of humans, plants, and animals in healthy conditions. When considering the human systems, they identify these critical dimensions: power relations; governance (community, regional, national, and international); prosperity in the form of adequate income for individual wellbeing and provision of social programs; equity in healthcare, education, early childhood programs, income security, safety, housing and employment; and mental, social and physical health. Brousselle and McDavid acknowledge that consideration of these natural and human system dimensions is not easy: “some dimensions are also difficult to assess, clear criteria and norms do not exist and too often some important goals are discarded because of the challenges measurement represents” (p. 10). Mixed methods researchers are in a position to contribute to designing research that addresses these complexities (for further discussion, see also Chapter 33, this volume). Even though no clearcut strategy exists for mixed methods researchers to take this courageous step, the literature reviewed in this chapter provides guidance on what to consider in developing new designs. The concept of quantitative and qualitative methods needs to be expanded to include the structuring of studies that incorporate building relationships, addressing power inequities and planning for sustainability.

A CALL TO ACTION The world is facing a climate crisis that is inherently linked with economic and social justice. Researchers can contribute to increased justice by considering these three forms of justice in the

design of their work. The use of a transformative lens provides guidance for designing mixed methods studies that extend thinking beyond the integration of quantitative and qualitative methods. Transformative mixed methods researchers need to develop strategies that can engage with a broad coalition of stakeholders in culturally respectful ways. The inclusion of those who hold traditional power and those who are vulnerable and marginalized is key to understanding the nature of the problem and feasibility of solutions. In order to increase the impact of the mixed methods studies, time spent on developing relationships and coalitions that consciously address power inequities is key. The transformative mixed methods researcher can learn from social activist strategies that have brought about change to fundamental social, economic and environmental issues. Being inclusive, ensuring effective communications throughout the study, building in strategies for sustainability and monitoring on-going program activities can be integrated into mixed methods studies to increase impact. Some social activist strategies may seem far-fetched to researchers, such as going on strike, boycotting businesses, and protesting at stockholder meetings. However, the idea of making problems visible in a very public way and finding ways to create impetus for change are associated with these actions. Change will come when researchers acknowledge power differences and design their studies to address those inequities. Having community members as co-researchers is one way that has been used to address power differences in culturally respectful ways (Flecha, 2014). Building coalitions that are representative of the full range of those impacted by the research is another strategy. However, researchers need to ask critical questions about the nature of the coalitions and their role in the research study. For example, who sets the agenda for the research? Whose voices are given privilege? How do coalition members influence understanding of culture and context? What role does the coalition play in developing the research questions and instruments? Who is engaged in the analysis, interpretation, and use of the data and findings? If the coalitions are engaged in meaningful ways, then they can play a role in sustaining the change and monitoring the system to see when and where revisions are needed. The use of a transformative mixed methods lens is not easy; however, the stakes are high if researchers choose to ignore the implications of their work for environmental, economic and social justice. Guidance is available to integrate consideration of factors related to these forms of justice in the transformative mixed methods literature reviewed in this chapter (see Love et  al., 2019;

Mixed Methods Designs to Further Social, Economic and Environmental Justice 57

Mertens, 2018a and 2018b; 2020a and 2020b; Mertens & Wilson, 2019) as well as in the work of environmental justice advocates (Brousselle & McDavid, 2020). What is needed is a commitment to action and a willingness to consider a different role for researchers, one that challenges an oppressive status quo and contributes to increased justice.

WHAT TO READ NEXT Widianingsih, I., & Mertens, D. M. (2019). Transformative research and the sustainable development goals: challenges and a vision from Bandung, West Java. International Journal of Transformative Research, 6(1), 27–35.

This open-source article describes the use of a transformative lens to address environmental, economic, and social justice in Indonesia. Mertens, D.M. & Wilson, A.T. (2019). Program evaluation theory and practice: A comprehensive guide (2nd ed.). Guilford Press.

This book provides detailed guidance on planning and implementing transformative studies. Chouinard, J.A. & Cram, F. (2020). Culturally responsive approaches to evaluation. Sage.

This book delineates how to conduct a culturally responsive study in national, international, and Indigenous settings.

REFERENCES Brousselle, A., & McDavid, J. (2020). Evaluation for planetary health. Evaluation, 1–16. https://doi.org/1 0.1177%2F1356389020952462 Chilisa, B., & Mertens, D. M. (2020). Indigenous Made in Africa evaluation frameworks: Addressing epistemic violence and contributing to social transformation. American Journal of Evaluation, 1–12. https://doi.org/10.1177%2F1098214020948601 Chouinard, J. A., & Cram, F. (2020). Culturally responsive approaches to evaluation. Sage. Cram, F., Chilisa, B., & Mertens, D. M. (2013). The journey begins. In D. M. Mertens, F. Cram, & B. Chilisa, B. (Eds.), Indigenous pathways into social research. Left Coast Press. Crisanti, A. S., Case, B. F., Isakson, B. L., & Steadman, H. J. (2014). Understanding study attrition in the evaluations of jail diversion programs for persons with serious mental illness or co-occurring substance disorders. Criminal Justice and Behavior,

41(6), 772–790. https://psycnet.apa.org/doi/ 10.1177/0093854813514580 Deitz, S., & Meehan, K. (2019). Plumbing poverty: Mapping hot spots of racial and geographic inequality in U.S. household water insecurity, Annals of the American Association of Geographers, 109(4), 1092–1109. https://doi.org/10.1080/2469 4452.2018.1530587 Dhaliwal, K., Casey, J., Aceves-Iñiguez, K., & DeanCoffey, J. (2020). Radical Inquiry—Liberatory Praxis for Research and Evaluation. New Directions for Evaluation (166), 49–64. https://doi.org/10.1002/ev.20415 Dobinger, J. (2021). Mainstreaming the environment into evaluations – can we walk the talk? Earth Eval. www.eartheval.org/blog/mainstreamingenvironment-evaluations-can-we-walk-talk Flecha, R. (2014). Using mixed methods from a communicative orientation: researching with grassroots Roma. Journal of Mixed Methods Research, 8(3), 245–254. https://journals.sagepub.com/ author-instructions/MMR Greenwood, S. (2019). Beaten down, worked up. Alfred A. Knopf. Hood, S., Hopson, R., & Frierson, H. (2015). This is where we continue to stand. In S. Hood, R. Hopson, & H. Frierson (Eds.), Continuing the journey to reposition culture and cultural context in evaluation theory and practice. (pp. ix–xvii). Information Age Publishing. Levin, L. (2020). Social justice: A contemporary challenge for social good. Research on Social Work Practice, 39(2), 186–195. 10.1177/ 1049731519854161 Love, B., Vetere, A., & Davis, P. (2019). Handling ‘hot potatoes’: Ethical, legal safeguarding, and political quandaries of researching drug-using offenders. International Journal of Qualitative Methods, 18, 1–9. https://doi.org/10.1177%2F1609406919859713 Lucero, J., Wallerstein, N., Duran, B., Alegria, M., Greene-Moton, E., Israel, Kastelic, S., Magarati, M., Oetzel, J., Pearson, C., Schulz, A., Villegas, M., & White Hat, E. R. (2018). Development of a mixed methods investigation of process and outcomes of community-based participatory research. Journal of Mixed Methods Research, 12(1), 55–74. https://doi.org/10.1177%2F1558689816633309 McBride, D., Casillas, W., & LoPiccolo, J. (2020). Inciting social change through evaluation. In L. C. Neubauer, D. McBride, A. D. Guajardo, W. D. Casillas, & M. E. Hall (Eds.), Examining issues facing communities of color today: The role of evaluation to incite change. New Directions for Evaluation, 166, 119–127. Mertens, D. M. (2018a). Mixed methods design in evaluation. Sage. Mertens, D. M. (2018b). Transformative mixed methods and policy evaluation. Diritto & Questioni Pubbliche, 18(1), 247–264.

58

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Mertens, D. M. (2020a). Research and evaluation in education and psychology (5th ed.). Sage. Mertens, D. M. (2020b). Transformation as a goal of mixed methods research in the Caribbean. Caribbean Journal of Mixed Methods Research, 1(1), 16–28. Mertens, D. M. & Wilson, A. T. (2019). Program evaluation theory and practice: A comprehensive guide (2nd ed.). Guilford Press. Miller, R. L. (2020). Reducing stigma and discrimination in access to HIV health care for gay and bisexual men and transgender women using mystery patients in Cameroon and Zimbabwe. Michigan State University. Mohai, P., Bryant, B., & Slatin, C. (2020). Commemorating, celebrating and rededicating to the fight for environmental justice: National environmental justice game changers. NEW SOLUTIONS: A journal of Environmental and Occupational Health Policy, 30(3), 213–225. https://doi.org/10.1177 %2F1048291120962147 Puigvert, L., Valls, R., Yeste, C. G., Aguilar, C., & Merrill, B. (2017). Resistant to and transformations of gender-based violence in Spanish universities: A communicative evaluation of social impact. Journal of Mixed Methods Research, 13(3), 361–380. https://doi.org/10.1177%2F1558689817731170 Rothstein, R. (2017). The color of law. Liveright Publishing Corporation. Slate R. N. (2003). From jailhouse to Capitol Hill: Impacting mental health court legislation and defining what constitutes a mental health court. Crime & Delinquency, 49(1), 2003, 6 ff.

https://citeseerx.ist.psu.edu/viewdoc/download?d oi=10.1.1.955.3797&rep=rep1&type=pdf Sordé Marti, T., & Mertens, D. M. (2014). Mixed methods with groups at risk: New developments and key debates. Journal of Mixed Methods Research, 9(3), 207–211. https://doi-org.login. ezproxy.library.ualberta.ca/10.1177%2F1558 689814527916 Spilsbury, M. (2020). Review of UNEP’s contributions to poverty reduction across environment-focussed evaluands. In UN Evaluation Group (Ed.) Environmental and Social Impacts: How to integrate them into evaluations? UNEP. November 17 2020. UN Office for Disaster Risk Reduction. (2020). Human cost of disasters: Review of the last 20 years 2000–2019. United Nations Office for Disarmament Affairs. Washington Post (2015). Justice for none. Diversion Books. Widianingsih, I., & Mertens, D. M. (2019). Transformative research and the sustainable development goals: challenges and a vision from Bandung, West Java. International Journal of Transformative Research, 6(1), 27–35. https://doi.org/10.1515/ ijtr-2019-0005 Wolfe, S. M., Price, A. W., & Brown, K. K. (Eds.) (2020). Evaluating community coalitions and collaboratives. New directions for evaluation, 165. Yeampierre, E., & Klein, N. (2020). Spotlight: A just recovery for Puerto Rico. In A. L. Plough (Ed.), Culture of Health in Practice (pp. 171–172). Oxford University Press.

5 Developments in Mixed Methods Designs: What Have Been the Dominant Pathways and Where Might They Take Us in the Future? Katrin Niglas

INTRODUCTION The use of mixed methods (MM) designs has gained immense popularity in recent decades. In the SAGE Encyclopedia of Research Design, Pinto (2012, p. 813) claims that “Most study designs today need to include both quantitative (QN) and qualitative (QL) methods for gathering effective data and can thereby incorporate a more expansive set of assumptions and a broader worldview.” While I am somewhat more modest about the prevalence of MM designs, it is both exciting and challenging to navigate the variability and plethora of approaches the methodologists and practising researchers from different continents and disciplines are bringing to the field. Building innovations on systematic and representative foundations helps to form a solid bridge to the future, as the Handbook aims. Therefore, the aim of this chapter is to provide an overview of some influential historical developments and a glimpse from these grounds into the possible future trends related to MM designs. Having agreed to the challenge of writing this chapter, my journey started with lots of enthusiasm and hope for personal academic growth. Serving in a full-time academic leadership position for more than ten years had limited my opportunities

to systematically follow the wealth of emerging MM literature, even though my passion for MM and connections, as well as a little input to the MM community, was carefully maintained. Being inspired by the open-minded approach and the principle of informed creativity, I was urged to take a broad view on MM designs through historical and international lenses. My earlier criticisms of some developments, including the overwhelming emphasis on fixed typologies of MM designs (Niglas, 2009, 2010), did not appear as an obstacle as I hold that the open-minded and creative approach is productive and sterling only if it is based on the knowledge about and understanding of the crucial issues and influences that have been and are forming the development of scholarship about MM designs. Being well aware of the disputes about the terminology and growing plurality of the concepts related to MM designs, which are common to the fields in their phase of fast development (Niglas, 2009; Bergman, 2011), I follow the call of the Mixed Methods International Research Association (MMIRA) task force that was established by the Executive Board of MMIRA in 2015 to cast an international gaze into the future of MM and use “mixed methods” as the general term for the tradition (Mertens et al., 2016a). Recognizing that most commonly MM are about combining

60

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

QL and QN methods or methodologies, I purposefully adhere to the core criteria of MM suggested in the above-mentioned report: “use of more than one method, methodology, approach, theoretical or paradigmatic framework” and “integration of results from those different components” (Mertens et al., 2016a, p. 4). In 2016, the task force report envisioned several “fertile areas” for the future development of MM research related to MM designs: • Fostering discussion on ways of conceptualizing MM designs without rigidly imposing a particular typology, but rather by delineating and defining dimensions on which they vary. • Developing MM approaches and designs that help to contribute solutions for important “wicked problems” and “grand challenges” the world is confronted with. That takes thinking beyond the concept of methodology to explore the meaning of a socially responsible role and critically examining the philosophical assumptions that guide their thinking. • The methodological repertoire of MM researchers could be widened by practice-oriented research designs like action research and design research, which would extend the range of problems we can tackle using MM, and especially those helping a socially responsible turn in MM research. • Promoting the adoption and adaptation of information technologies in order to widen the repertoire of research methods. New digital tools facilitate the emergence of MM designs for complex research problems and settings through helping to make connections between parts of a MM design. After six years of the report being released, it is intriguing to ask if these important trends have been nurtured by the members of the MM community and continue to be actual in the coming years? However, we should not forget that the task force called the MM community to recognize that MM are practised in many different ways, under different names, in different disciplines and countries. Therefore, in the search for new pathways advancing our methodological practices, it is important to scrutinize the historical influences on the developments and implementation of MM designs enabling learning from and being inspired by the existing knowledge and experiences, but also avoiding some struggles and logjams. As a modest contribution to the input, the Handbook provides for advancing our understanding of the essential issues related to MM designs.

In this chapter, I first focus on the developments of the notion of design in the context of MM research, and then map the key influences that shape our understanding of the MM designs. Although I seek for (and invite you to join) an inclusive and open-minded mental journey, the current historical overview and glance at the latest developments are inevitably somewhat superficial as being limited to the publications in English and fitted into the word limits of an article. Interested readers can find some suggestions for further reading at the end of the chapter.

KEY TERMINOLOGY: THE VEX ABOUT THE CONCEPT OF DESIGN The word “design” can be used in the research context both as a verb focusing on the process of designing a study and as a noun designating the result of this process (Schoonenboom & Johnson, 2017). Generally, the term “research design” refers to the logical framework or overall strategy that integrates different methodological components of the study in a coherent manner. Revisiting the definitions of MM research and browsing the literature about MM designs indicates that the use of the notion “design” is varied and multi-layered in the context of MM research (see Table 5.1 and Figure 5.1). Quite often, the terms “MM research” and “MM design” have been used as synonyms (e.g., Pinto, 2012), or MM research is defined as a specific type of research design. This kind of alignment is typical when the focus of MM research is put primarily on combining QN and QL data in a single study or a set of related sub-studies. It is important to notice that in this general meaning, “MM design” is commonly used as a singular term and distinguished from QN and QL research, or pure QN design and pure QL design. This most general conceptualization of MM design, comprising discussions and disputes not only on the level of methodologies and methods, but also on the level of worldview positions and theoretical assumptions, is illustrated as the outer layer in Figure 5.1. However, one has to notice that QN, QL and MM as three methodological schools of thought or clusters of research are not internally uniform in actual research practice, but rather big families comprising various traditional (e.g., experiment, ethnography, etc.) or typical MM (e.g., sequential explanatory, triangulation, etc.) research strategies or methodologies which we are more commonly used to conceptualize as research designs

DEVELOPMENTS IN MIXED METHODS DESIGNS

61

Table 5.1  The varied use of the notion “design” within the context of mixed methods research Concept

Definition

Clarification or examples

Research design

Logical framework or overall strategy that integrates different methodological components of the study in a coherent manner. As such, research design includes a succinct and logical plan or description of how one tackles established research question(s) through the collection, interpretation, analysis, and discussion of data.

MM design as opposed to QUAN and QUAL design

In this context, MM design is used to denote the whole family of MM research designs and often equalized with MM research.

MM design as one of many available MM design types

MM design denotes the overall logic and the set of principles of how the (QN and QL) aspects or phases of the research process are executed, but also guiding the choice of methods and techniques in concordance with the objectives of research study.

MM design as related to multimethod design

Sometimes “multimethod designs” are defined as designs combining two or more methods or methodological approaches from the same tradition referring to the QN and QL approaches (Creswell & Plano Clark, 2007; Schoonenboom & Johnson, 2017). Other more flexible stances allow a combination of any different methodological approaches (QN and QL), and emphasize the independence or the phase, and the ways of integration of these combined approaches (Bazeley, 2018; Johnson et al., 2007; Morse, 2003).

In social sciences research, obtaining information relevant to the research problem entails specifying the type of evidence needed —for example, to test a theory, to evaluate a programme, or to accurately describe and assess meaning related to an observable phenomenon. The function of a research design is to ensure that the evidence obtained enables the researcher(s) to address the research problem effectively and find answers to research questions as logically and as unambiguously as possible (de Vaus, 2004). “MM research is a research design with philosophical assumptions as well as methods of inquiry. …” (Creswell & Plano Clark, 2007, p. 5). Definitions of MM research in Johnson et al. (2007): “A mixed methods design is a plan for a scientifically rigorous research process . . . ” (Janice Morse). “Mixed methods research is a type of research design . . . ” (Abbas Tashakkori & Charles Teddlie). “Mixed methods research is a research design (or methodology) . . . ” (John Creswell). Operating on that level, the term “research strategy” can sometimes be used as a synonym for “research design” (e.g., Creswell, 2003; Piccioli, 2019). In the early writings of some authors like Creswell and Plano Clark (2011), MM designs in the latter meaning have also been referred to as “variants” or “models”. Thus, the relationship of MM designs and multimethod designs is variably conceptualized. Most commonly, the multimethod design is taken either: a) as the subtype of MM designs where the integration of combined methods or methodologies remains limited, or b) as akin to MM designs with the difference that methods and methodologies combined come from one (QN or QL), but not from both traditions.

Source: Author created.

62

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 5.1  The conceptual map of the notion “design” as used in the context of (mixed methods) research Source: Author created.

(Bergman, 2011). My preference is to use the concept “MM design” on the latter methodological level where “design” is denoting the overall logic and the set of principles of how the phases of the research process are executed, but also guiding the choice of methods and techniques in concordance with the objectives of a research study. As emphasized in Figure 5.1, I point out that on this second methodological level, there is variability in how the authors use and relate concepts of “mixed methods designs”, “multiple-methods designs”, “multimethod designs”, etc. While in the early writings, “multiple-method design” was used as an umbrella term for all types of combined design models (Tashakkori & Teddlie, 2003), it is discussed above that now the term “mixed methods” is generally recommended for this purpose (Mertens et  al., 2016). However, the relationship between the concepts of MM and multimethod designs remains varied in the methodological literature (see Table 5.1).

To make things around the concept of “design” within the MM research context even more complicated, one can find constructions where the MM design is described through research stages or “design components”. The most commonly listed design components are (see, for example, Johnson et al., 2007, p. 123; Maxwell, 2013; Niglas, 2009; Piccioli, 2019): • • • •

paradigmatic/conceptual/theoretical framework; research goals/questions/hypotheses; research strategy/design; methods for sampling, data collection and analysis; • validation and inference techniques. In Figure 5.1, the design components are located in the inner layer as these are the methodological core elements of any research study. Notice, that “design” is sometimes listed as one of these

DEVELOPMENTS IN MIXED METHODS DESIGNS

components or elements of the overall MM design (Headley & Plano Clark, 2020). Even though it may seem tautological in the first place, it does make sense to think in terms of traditional QN and QL research designs like experiment, survey, ethnography or case study design, and recognize that within a study with MM design, one can combine and integrate these more traditional research designs or embed data from another tradition to the chosen traditional design. Incorporating MM procedures into traditional designs, such as case studies, formative evaluation in experiments, and action research, was emphasized as the emerging trend by Creswell (2009) in his editorial to the Journal of Mixed Methods Research (JMMR) and has been elaborated since then (e.g., Creswell & Plano Clark, 2018; Plano Clark & Ivankova, 2016). As a special feature of MM designs is the combination of various methodological aspects within and/or between the research stages, MM designs are commonly described or defined by a set of design dimensions (or characteristics) like timing or priority given to the QN and qualitative components (Creamer, 2018; Greene et al., 1989; Guest, 2013; Plano Clark & Ivankova, 2016). The set of design dimensions is helping to characterize the prevalence and/or relationship between design components incorporated into a particular MM design from different methodological traditions, and is therefore the mediating layer between design components and types of MM designs in Figure 5.1. Shoonenboom and Johnson (2017) give great importance to these dimensions and, drawing on the work of many earlier writers, list seven primary dimensions and ten secondary dimensions of MM designs. These include, for example, the purpose of mixing; timing (comprising both simultaneity and dependence); point of integration; typological vs. interactive and planned vs. emergent design; complexity of the mixed design; degree to which the participants, researchers in the research team and/or methods and techniques used are similar or different, etc. Disputes about an appropriate terminology are part of any discipline (Bergman, 2011). This seems to be especially true about the field of MM designs which has been metaphorically compared to a phoenix due to its rapid development (Cameron & Miller, 2007). Considering this, we should not become exhausted in our search for systematic but flexible conceptualizations and frameworks helping to understand and cope with the inevitable pluralities. I hope that the above discussion and the schematic model in Figure 5.1 draws together some important discussions on ways of conceptualizing MM designs that were fostered by the MMIRA task force report (Mertens et al.,

63

2016) and will help to alleviate the vex about the use of the concept “design” in the context of MM research.

KEY INFLUENCE: WHAT SHAPES OUR UNDERSTANDING OF MM DESIGNS? Even though the early debates about social science methods can be dated back to the seventeenth century and the combined use of QN and QL methods to an even earlier date, MM as an identifiable methodological movement has a short history that can be traced back to the 1980s (Tashakkori & Teddlie, 2003). It is common to cite the conceptual work of Bryman published in 1988 and Greene et al., published in 1989 as the first attempts to classify MM designs (Mertens et  al., 2016; Molina-Azorin, 2018). Let us, thus, take the 1980s as a baseline for the emergence of the systematic approach to MM designs and the self-identified community of MM researchers, but be aware that this work is preceded by a solid history of practices integrating QL and QN methods in various disciplines of academic research. This early development of the “design integration” is comprehensively reported in several writings by Joseph Maxwell (Maxwell, 2016; Maxwell et al., 2015; see also Chapter 3, this volume). In the following, I will trace the key influences that have fertilized the development and implementation of MM designs by building grounds for respectful discussion on various conceptualizations of MM designs, and widening the scope of our research focuses and methodological repertoire as envisioned by the MMIRA task force (Mertens et al., 2016). These historical key influences or dominant pathways are plotted in Figure 5.2. My vision is that in the years to come, we will witness an ever increasing fuse of these influences, as the pathways will cross and mingle, helping thereby to enrich our research practices. But let’s turn back to the roots for a moment. Research in the first half of the twentieth century was largely QN, and the second half of the century witnessed a rapid sophistication of that methodological tradition (Lund, 2012). This “technocratic turn” evoked protests in some research communities and triggered the search for alternative ways of studying social realities. In parallel with the evolution of a strong QL research community and practices around the 1970s, several scholars with a mainly QN background from the US, the UK and some other English-speaking countries laid the foundation for MM research largely as a critique against QN research which had dominated health,

64

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 5.2  The historical key influences on the development of mixed methods designs Source: Author created.

education, economic, educational and social science methods for several decades. Considering these roots of MM research and the swelling disagreement between QN and QL methodologists on the philosophical foundations of research methodology—sometimes referred to as the “paradigm wars” (Gage, 1989; Hammersley, 1992)—it is not surprising that for the purposes of legitimation and practical considerations, the MM research community quickly turned to the search for a systematic typological representation of MM research designs (e.g., Creswell, 2003; Greene et  al., 1989; Tashakkori & Teddlie, 1998). I will label this period from the end of 1980s until the mid-2010s an “era of MM design typologies”. The heavy emphasis on incommensurable paradigms and perceived over-systematization of MM designs that was often taken as not accurately capturing the plurality of the actual MM research practice, and therefore limiting the potential of MM research, triggered several MM researchers and methodologists to question the usefulness of fixed typologies and propose more flexible frameworks for conceptualizing MM designs (e.g., Guest, 2013; Maxwell & Loomis, 2003; Niglas, 2009). The above central branches of development have been assisted by the bulk of MM prevalence studies mapping the use of MM designs and various design issues of the actual MM research practices (Molina-Azorin & Fetters, 2016). In Figure 5.2, these are noted as prevalence studies focused on MM designs. From the mid-2010s, some assimilation and the gradual acceptance of a more flexible approach by the authors of initially fixed typologies can be witnessed (see, for example,

Creswell, 2015a). However, the representation of the MM design typologies in popular educational texts about the MM research methodology has become vast, and therefore strongly influences the young generation of researchers around the globe. On the crossroads of typologies and flexible frameworks of MM designs, many authors have recently described particular innovative MM designs, which are well fitted to meet the complex and wicked problems of the contemporary social world (e.g., Bamberger, 2016; Headley & Plano Clark, 2020; O’Halloran et al., 2018). And finally, as Maxwell (Chapter 3, this volume) argues, the integrated designs have been continuously used and developed in the research communities of various disciplines who are not (yet) in communication with the “self-identified ‘mixed methods’ community” and do not relate their work to any MM design typologies. Let me call this an intuitive line of development in the MM design principles and practice (“intuitive MM practice” in Figure 5.2). It is of no surprise, though, that this line has given valuable inspiration to the MM community in conceptualizing and designing MM studies.

MM DESIGN TYPOLOGIES: IDEAS IN CONSTANT DEVELOPMENT In parallel with the emergence of the self-identified community of MM researchers, the attempt to plot and label systematically the possibilities

65

DEVELOPMENTS IN MIXED METHODS DESIGNS

for various ways of combining the QN and QL approaches in the endeavour of empirical research resulted in a number of early classifications of MM research designs (see Table 5.2). An exhaustive overview of these initial MM design typologies was provided in my dissertation with some critical notes pointing to the need for further academic discussions and considerations (Niglas, 2004). Although emphasizing different qualities in MM designs and varying in the level of detail, typologies are based on some common methodological characteristics and procedural features being, thus, commonly built as logical combinations of chosen design elements and/or dimensions. In some typologies, also the purpose of mixing QN and QL approaches is taken as a basis for classification (e.g., Greene, 2007). As a typical example of early design considerations, one can think of timing or sequence as one of the dimensions describing the combination of QN and QL approaches or methods with possible values being simultaneous and sequential. Let us take the priority or weight as a second dimension with possible values being equivalent and dominant. For the dominant status, one can further define the possibilities like qualitatively driven and quantitatively driven. These dimensions and their possible combinations yield several different MM design

types like equivalent status simultaneous design or quantitatively driven sequential design, which can be represented using the popular notation system proposed by Morse (1991) as follows: QUAL + QUAN QUAN → qual The usefulness of MMD typologies is often expressed as initially proposed by Teddlie and Tashakkori (2009), and Creswell and Plano Clark (2011), including arguments such as: • Establishing a common language in an emerging field; legitimizing the field; conveying legitimacy to new audiences. • Broad conceptual classification is important to guide our thinking, especially for the novice; outline the design features common to a group of MM studies; useful pedagogical tool. • Guiding practice; providing tools for designing studies; can help make more informed decisions when choosing an approach for designing their study in response to a specific research question. • Generating new possibilities. In parallel with a valuable contribution to the development of MM designs, the growing number

Table 5.2  Examples of early classifications of studies by their ways of using/combining quantitative and qualitative approaches (adapted from Niglas, 2004) Proposed classification

Patton, 1980 Mark & Shotland, 1987

Bryman, 1988 Brewer & Hunter, 1989 Creswell, 1995

Tashakkori & Teddlie, 1998

Pure designs Purely quantitative or purely qualitative designs (may involve the use of several data sources and/or data-gathering instruments from the same approach). Quantitative study; qualitative study Quantitative study; qualitative study Quantitative study; qualitative study Monomethod studies Quantitative study; qualitative study Monomethod studies

Combined designs Multimethod designs Mixed designs Designs where both Designs where elements quantitative and of quantitative and qualitative approaches qualitative approach are are used, but they remain combined in various ways relatively independent within different phases of until the interpretation the study. stage. Triangulation Mixed-methodology design Triangulation;* bracketing model;* complementary multiplism Ten different ways of integration Multimethod studies Two-phase design; dominant-less dominant design Mixed method studies

Methodological hybrids Composite method studies Mixed-methodology design

Mixed model studies

* These models can be used within purely quantitative or qualitative studies as well. Source: Adapted from the author’s earlier publication (Niglas, 2004) where the copyright belongs to the author.

66

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

of different typologies brought with it the inevitable implication of making the terminology and the concepts proposed to define and frame MM designs increasingly more varied, and therefore somewhat difficult to navigate (Niglas, 2009; Chapter 2, this volume). Critical issues related to the typology-driven approach to MM designs were acknowledged by the MM community and summarized by Teddlie and Tashakkori (2003, p. 32) in the first MM Handbook, pointing out that: • there is a growing number of typologies of MM research designs; • none of these typologies are exhaustive; • typologies vary by the criteria that are used to distinguish among the research designs; • in some cases, the researcher may have to develop a new MM design because none of the existing designs is best for his or her research project. The latter provoked the attempt to review the existing typologies, whereby many authors focused on standardizing research designs in order to organize and simplify complex phenomena for cognitive, organizational, and communicative purposes developing, thereby, typologies to set MM designs within an inclusive framework (Mertens et  al., 2016; Piccioli, 2019). This process culminated (but did not end) in 2011 when Creswell and PlanoClark proposed the system of “Prototypes of Mixed Methods Designs”. Soon after, the “move toward the simplification” (Guest, 2013, p. 145) of MM typologies by summarizing and reducing the design dimensions was called for and exercised by several typology-driven authors. It was emphasized that some of the proposed designs are overly complex and researchers were called to return to more simple MM designs with the option of seeking out variants of them if required (Creswell, 2015a). Looking into the future of MM research more than five years ago, the MMIRA task force and some other authors stated that consensus does not exist among scholars how best to describe and delineate the major MM designs and what criteria should be considered in guiding the choice of design, and thereby envisioned that MM designs remain one of the most debated topics in the MM research literature (Mertens et al., 2016; Plano Clark & Ivankova, 2016). As early as 2009, Creswell, in his editorial to the JMMR, acknowledges that the proposed types of MM designs are not complex enough to mirror actual practice, and reflected that he needed to rethink how to look at the designs for MM researchers (Creswell, 2009). Although it is widely accepted today that typologies are not exhaustive and provide only ideal exemplary design types, while in actual research practice, new innovative (and complex)

MM designs are constantly created, many members of the MM community hold that typologies are useful in guiding researchers and helping them to make more informed decisions when designing their MM studies (Tashakkori et al., 2020). Thus, the design typologies are continuously scrutinized and renewed or supplemented by more advanced designs (e.g., Creswell, 2015b; Creswell & Plano Clark, 2018; Ivankova, 2015; Johnson & Christiansen, 2017; Morgan, 2014; Tashakkori et al., 2020). Looking from the bird’s-eye view at the most recent MM design typologies in Table 5.3, I echo Creswell and Plano Clark (Chapter 2, this volume) that a certain move towards the conceptual and terminological convergence in core concepts of typologies is noticeable. At the same time, the clear vision is that the mainstream MM design typologies will be depicted as ever more flexible frameworks and enriched by the examples of complex applications of MM designs in various research fields and contexts (e.g., Mertens, 2018). The generic nature of the current chapter does not leave room for a detailed overview of the wealth of current discussions and innovations related to the typologies of MM designs, but an interested reader can find useful summaries in several articles and textbook chapters published by the members of the MM community (e.g., Cameron, 2009; Mertens, 2018; Plano Clark & Ivankova, 2016; Schoonenboom & Johnson, 2017; Tashakkori et al., 2020; see also Chapter 2, this volume).

EDUCATIONAL TEXTS ON MM DESIGNS: THE PREVALENCE OF TYPOLOGIES While researchers from many different countries have been among the founders of the MM tradition and engage in critical discussions about various issues of MM research, including MM designs (Creswell & Sinley, 2017), it is notable that the typology-driven approach to MM designs has been developed almost exclusively by scholars from the US. Regardless of this bias, the typology-driven approach to MM designs has a remarkable global influence and is shaping MM practices around the globe. In their overview paper of international developments, Creswell and Sinley (2017, p. 101) emphasize the importance of training and the translation of MM textbooks in order to support the “global expansion of MM”, pointing out that several textbooks are translated into multiple languages other than English. Notably, in these (and other potential textbooks to be translated), the typology-driven approach to MM designs is prevalent.

DEVELOPMENTS IN MIXED METHODS DESIGNS

67

Table 5.3  Selected most recent typologies of mixed methods designs The authors

MM designs in the proposed typology

Plano Clark & Ivankova (2016)

Basic designs Concurrent QN + QL design Sequential QN + QL design Sequential QL + QN design Intersecting basic designs with other approaches MM experiment MM case study MM evaluation MM action research Transformative MM research Concurrent designs Equal status concurrent design (QL+QN) Qualitatively driven concurrent design (QL+quan) Quantitatively driven concurrent design (QN + qual) Sequential designs Equal status sequential designs: (QL -> QN) and (QN -> QL) Qualitatively driven sequential designs: (QL -> quan) and (quan -> QL) Quantitatively driven sequential designs: (QN -> qual) and (qual -> QN) Concurrent (QN and QL data collection in parallel) 3 subtypes with QN, QL or equal priority and concurrent/sequential timing of data analysis Sequential exploratory (QL data collection first) 3 subtypes with QN, QL or equal priority and concurrent/sequential timing of data analysis Sequential explanatory (QN data collection first) 3 subtypes with QN, QL or equal priority and concurrent/sequential timing of data analysis Iterative or multiphase (QN and QL data collection) 3 subtypes with QN, QL or equal priority and multiphase data analysis Core designs Explanatory sequential design Exploratory sequential design Convergent design Complex MM designs Core MM designs can be intersected with other research approaches or frameworks resulting in a variety of complex designs. The authors describe four exemplary types of complex MM designs: MM experimental (or intervention) design MM case study design MM participatory – social justice design MM evaluation design The authors use general typology comprising monomethod designs, multimethod designs and mixed methods designs Popular MM designs Conversion MM designs Parallel MM designs Sequential MM designs Hybrid MM designs

Johnson & Christiansen (2017)

Creamer (2018)

Creswell & Plano Clark (2018)

Tashakkori et al. (2020)

Source: Author created.

For the purposes of getting a glimpse into the current trends that the general educational texts, which are introducing the concept of MM designs,

are providing, I formed an opportunistic sample of ten English language methodological texts by the keyword search from the sources (including digital

68

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

sources) available in the library of Tallinn University (Estonia). Appendix 1 provides a list of the texts included in the overview sample. Considering that English is the most commonly spoken foreign language and that the methodological texts of influential academic publishers like Sage are typically the ones the university libraries are gaining access to, I suggest that my sample is rather close to those we would gain from the libraries of many countries where English is widely spoken. A simple analysis showed that all ten texts were introducing one or more MM design typologies with limited reference to the alternative possibilities for MM design conceptualization. In these texts, authors often summarize the situation similarly to Doyle et  al. (2019) who point out that a plethora of MM designs and typologies have emerged, which can be confusing for both novice and experienced MM researchers. On these grounds, the authors continue by providing a compact overview of one, or sometimes a few existing frameworks, pointing out that basic designs provide a sound starting point for those unfamiliar with MM designs. Most commonly introduced typologies in my sample were those proposed by Creswell and Plano Clark (2007, 2011), followed by Teddlie and Tashakkori (2003, 2009), and Greene (2007), Greene and Caracelli (1997), and Greene et  al. (1989). See Appendix 2 for a list of the typologies that were mentioned in the educational texts included in the overview sample. Without being able to provide systematic empirical evidence, my practical experience as an educator and reviewer for several academic journals indicates that, along with the growing popularity of MM research, these basic designs are often employed by novice researchers from many countries around the world.

PREVALENCE STUDIES FOCUSED ON MM DESIGNS: LEARNING FROM PRACTICE FOR WIDENING THE HORIZONS In the late 1990s and early 2000s, the growing popularity of MM research and ongoing methodological debates prompted authors from various research fields to scrutinize and map the actual practice of MM research. The ground-breaking study by Greene et al. (1989), where they reviewed 57 mixed-methods evaluation studies in order to enrich their conceptual work with an interplay of theory and practice, was followed by the rapidly emerging body of content analyses analysing the use of MM in a variety of academic fields (Creamer, 2018).

In my early small-scale study, articles from four volumes (years 1997–1999) of The British Educational Research Journal were reviewed. A total of 46 research articles were identified which reported at least some results of empirical investigation. These studies were classified on the basis of design elements and 31 per cent of them were considered MM designs and further divided into two clusters (Niglas, 1999). One of the clusters consisted of case studies with QL research strategy and non-random sampling techniques, but implementing mixed or mainly QN data and analysis methods. The other cluster consisted of experiments and small-scale surveys which integrated some QL analysis either on the basis of interview data or answers to open-ended questions in questionnaires. MM designs that were implemented appeared to be quantitative-driven, and there was seldom any reference to the emerging tradition of MM designs. Numerous later studies have pointed out the similar trends in the early use of MM designs. The literature review by Caruth (2013) summarizes several studies analyzing the implementation of MM designs. This early overview indicates that the adoption of MM designs in different academic fields was not uniform, but generally MM designs were gaining acceptance and becoming increasingly popular in actual research practice. On the other hand, it was common that the authors of empirical studies combining QL and QN approaches did not specify MM design with the exception of papers published in the JMMR (Parylo, 2012). On the contrary, the later analysis by Archibald et  al. (2015) indicated that approximately half of the researchers utilizing MM practices within the overarching framework of QL research self-identified their study as MM research and explicitly identified fixed MM design typology exploited. This provides some evidence that the evolving typological approach to MM designs had gained acceptance by the wider research community. The most fervent implementation of MM designs in the early period of MM developments is apparent in various subfields of education, including educational evaluation studies. For example, Ross and Onwuegbuzie (2010) report that the prevalence of studies with MM design in the Journal for Research in Mathematics Education (JRME) between 1999 and 2008 was approximately one third. Similar results are reported by Parylo (2012) in the field of professional development and in several more general overview studies of educational journals (Alise & Teddlie, 2010; Niglas, 1999, 2004; Truscott et  al., 2010). The prevalence of MM designs in the other fields did not exceed 5 per cent and several authors pointed

DEVELOPMENTS IN MIXED METHODS DESIGNS

out that the great potential of MM designs in their field has not yet materialized, calling for more systematic and extensive training of the research community. For some examples, you can see Frels and Onwuegbuzie (2013)—counselling; Gambrel and Butler (2013)—marriage and family therapy; Lopez-Fernandez and Molina-Azorin (2011)— behavioural sciences; Octlund et al. (2011)—nursing; Wisdom et al. (2012)—health. Few studies were located reporting the reviews of MM designs in actual research practice in the countries outside the mainstream English-spoken regions. Ngulube (2010) explored the library and information science journals in sub-Saharan Africa and found that MM designs are not prevalent, as historically the training of the researchers has not provided proficiency in both QN and QL research methods, and there is no tradition of mutual cooperation between the researchers from different methodological schools of thought. Zhou and Creswell (2012) delineate the MM practices by Chinese scholars in East China, summarizing that the local research community is in its early stage of utilizing MM designs. Mainly simple convergent parallel and sequential MM designs were implemented within the framework of health studies, social and educational research. The authors suggest that researchers from these countries should be more open to the international MM community, visit actively MM conferences, and communicate with experienced MM researchers. The predominance of the QN tradition in actual MM practice has been reported in a number of prevalence studies focused on MM designs. For example, Bryman’s (2006) content analysis of MM research articles identified the dominance of the components of QN approach in the majority of MM designs. Creamer’s (2018) analysis showed that in approximately two-thirds of MM designs, the priority was given to the QN strand or methodology. However, in some fields like education and family science, the studies with equal weight or QL priority were equally represented. Given the historical dominance of QN methods in various fields of social sciences, education and health-related studies, the above tendency can be taken as a natural pathway to the well-balanced implementation of QN and QL traditions within MM designs. But I call us as a community of MM researchers and educators to recognize that some of the popular MM design typologies have a built-in QN priority or dominance and the strictly systematic approach driving all-inclusive typologies is intrinsically compatible with the QN tradition, but not so inherent to the QL ways of doing research. Thus, for promoting qualitatively driven MM approaches, one has to notice that it may be difficult to state up front the exact design and that

69

it is sometimes imperative that researchers not allow their project to be a design-driven enterprise whereby a particular prior MM design template dictates in advance their data collection and analytical procedures (Hesse-Biber, 2018). A further critical issue evident in the results of prevalence studies is that the full potential for integration is rarely exploited (e.g., Bryman, 2006; Niglas, 2004). Bazeley (2018) argues that integrated designs constitute the heart of inquiry, but the challenge of integration is still not cleared up for MM researchers and becomes evident when MM studies are reported in a dissertation, report or article. Challenges to integration experienced in social, behavioural, and health sciences arise from the paradigmatic differences in philosophical orientation and disciplinary traditions associated with different methods, personal predispositions and preferences of researchers, the training and skills of researchers, and conflicts within mixed-background teams. Pressure from funding agencies for particular style of research limits opportunities. Journal preferences and restrictions on word limits further contribute to difficulties in reporting and publishing results from an integrated study. (Bazeley, 2018, p. 8)

Creswell and Sinley (2017, p. 96) point out the limiting influence of the historically rooted “methodological orientation” on the successful implementation of MM research and suggest on the basis of the interviews with MM researchers from eight countries that “these preferences certainly seem to exist within disciplines and perhaps within regions of countries”. Tracing the issue back to the lopsided training the senior scholars have received in different disciplinary and regional contexts, they argue that “it is imperative that researchers are open to discussing new research opportunities” that help best to serve the needs of growingly complex and transdisciplinary research problems. To reach these widened horizons for MM designs, we should find ways for alleviating the limiting expectations of funders and publishers.

ALTERNATIVES FOR MM DESIGN TYPOLOGIES: PERSISTENT CALLS FOR FLEXIBILITY AND INNOVATION The fixed nature and plurality evident in numerous typologies and taxonomies of MM designs have been persistently criticized by a number of

70

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

influential MM authors as failing to capture the actual diversity of MM studies (e.g., Bryman, 2006, 2007; Greene, 2007; Guest, 2013; Maxwell & Loomis, 2003; Maxwell et  al., 2015; Niglas, 2004, 2010). They argue that no typology can capture all possible variations in designing MM studies and in real research practice, MM research designs can be infinitely variable, depending on the immediate purposes and questions the researchers set. For example, Guest (2013, p. 141) states that fixed classification systems can be “useful for simple and less fluid types of MM research, but are not capable of capturing the complexity and iterative nature of larger, more intricate research projects”. He has presented a framework which places the integration of QN and QL data within the study (called points of interface) into the central role and operates with two descriptive dimensions—the timing and the purpose—which help to characterize these incidents of data integration. Concentrating on the points of interface between data sets instead of being imposed to classify on the level of the entire study “would lessen the challenges of trying to squeeze the complexity and fluidity of many projects into fixed research design boxes and lexicons” (p. 150). Also, the assumption that research designs are tightly bound to incommensurable methodological paradigms was challenged. Building on the concept of the researchers’ mental model introduced by Greene (2007), I have argued that the paradigmatic stance on research practice, including MM designs, has gradually given place to the conceptualization of research methodology as a multidimensional continuum (Niglas, 2010). This model is based on the presumption that the best understanding of different possibilities to compile a design for a piece of empirical research can be achieved through an open and creative, but at the same time a systematic and organised view on relationships between different methodological approaches and aspects of design. The latter presupposes the ‘deconstruction’ of the design of an empirical study into methodological aspects and scrutinising these aspects in the light of numerous continua defined by polar properties like numeric– narrative, preplanned-emergent, inductive–deductive, objective–subjective, value neutral – value rich, etc. (Niglas, 2010, p. 231)

Thus, the proposed alternative to the typologybased approach to research designs calls for pouring over the logic of more traditional research strategies (e.g., survey, experiment, case study, ethnography, action research, etc.) and deconstructing research designs into methodological

aspects, understanding that there is no one-to-one relationship between the methodological choices one can implement within different levels of the design. This means that there is a possibility to combine QN and QL elements within or between any design components of the study if this will help to get a more adequate and plausible answer to the research questions (Niglas, 2009). The latter idea is in line with the interactive model of design proposed by Joseph Maxwell (Maxwell, 2013; Maxwell & Loomis, 2003; Maxwell et al., 2015; see also Chapter 3, this volume), which appears to be the most often cited alternative approach to MM typologies. These and numerous other alternative stances (e.g., synergistic approach by Hall and Howard, 2008; holistic integrationism by Plowright, 2011, etc.), which can’t be introduced here due to the limited space, have pioneered the systematic but open and flexible approach to MM designs, paving the way to the MM practices serving the complex challenges of our societies. In parallel, practising researchers started to publish examples illustrating how available MM designs were not congruent with the purposes of their particular study and scrutinizing complexities involved in making decisions about and the implementation of hybrid MM design (e.g., Vrkljan, 2009). On these grounds, a series of specific new MM designs started to emerge, promoted by the authors of diverse methodological and regional backgrounds (e.g., Miller & Fredericks, 2006; Morse, 2010; Morse & Maddox, 2014). The JMMR, the International Journal of Multiple Research Approaches (IJMRA) and both volumes of the Handbook of Mixed Methods in Social & Behavioural Research edited by Tashakkori and Teddlie (2003, 2010) have provided prosperous possibilities to focus on MM designs well applicable in specific disciplinary contexts (e.g., inclusive participatory designs for health sciences introduced by Kroll, 2011) or setting the traditional methodologies (e.g., longitudinal designs) and new technical possibilities (e.g., novel software tools) into the framework of MM designs. You can also find useful ideas from IJMRA, Vol. 1 (2), “Conducting longitudinal research”; Vol. 7 (2), “Mixed methods in genders & sexualities research”; or Vol. 5 (1), “Mixed methods research in the health sciences”, edited by Australian researchers; or Vol. 2 (1), “Computer assisted multiple and blended research”, edited by researchers from Switzerland and Italy. Early in the 2000s, authors from the UK and European countries like Germany, Austria, Belgium, Italy and Spain edited several books with a focus on innovative research approaches, including mixed and multi-method designs (e.g.,

DEVELOPMENTS IN MIXED METHODS DESIGNS

Bergman, 2008; Gorard & Taylor, 2004; Mayring et al., 2007; Rihoux & Grimm, 2006; van Meter, 2003). In these volumes, the authors go beyond the quantitative–qualitative divide, introducing, for example, the concept of design studies in the context of educational research, heuristics for stepwise comparative research design in the field of policy analysis, and numerous data-related methods and techniques that inherently integrate the features of QN and QL traditions. Lately, Poth (2018b, XV) has provided a thorough introduction to complexity-sensitive approach to MM research by illustrating six adaptive practices to deal with the “varying conditions of complexity” surrounding real-world research. She calls MM researchers to diagnose complexity in research conditions and to frame intentions of complex MM problems, paying attention to social, interpersonal and personal contexts, putting emphasis on describing and developing the capacity of MM design integrations, and generating solid evidence of complex MM research outcomes. Thus, in the context of complexity-sensitive MM research, adaptive mindset and integrative thinking on the one hand, and the courage to allow creativity in designs and to aspire for authentic reporting on the other, will allow a MM researcher to “be an innovator” (Poth, 2018b, p. 296). Finally, as Joseph Maxwell (Chapter 3, this volume) argues, a non-typological but explicit development of MM research designs occurred already in the first half of the twentieth century in many academic fields, but has been rarely recognized in the mainstream MM literature. This vast and continuous development of research designs integrating QL and QN methods that has taken place outside and independently of the conceptualizations provided by the self-identified MM community (I have referred to this line of development as an “intuitive MM practice” in Figure 5.2) should be recognized and used as a valuable source of inspiration for innovating MM practices.

IMPLICATIONS: A GAZE INTO THE FUTURE My historical overview commenced with a reference to the MMIRA task force’s recent gaze into the future of MM international developments (Mertens et al., 2016). The task force considered it important to learn from the history and implementation of MM in order to understand and benefit from the many different ways that MM has been and can be practised in different disciplines and countries. In this chapter, I have plotted some

71

central development lines of MM designs concerning terminology and key influences, hoping that a systematic and inclusive outline will help the reader to better steer through the complex set of ideas and inspire fruitful MM practices in the future. Following the historical pathways, it became apparent that although for a novice in MM research, the more restricted and simple design typologies are often considered to be a feasible and appealing starting point. The authors of more influential educational texts introducing one or the other typology of MM designs note the many alternatives, call for flexible thinking and pose a list of critical questions the researchers need to tackle when designing their MM studies (e.g., Plano Clark & Ivankova, 2016). It is also evident that the authors of initially fixed and all-inclusive design typologies have taken a turn towards more flexible approaches (see, for example, Chapter 2, this volume). These trends continue to encourage the MM researchers to adapt and adjust typical designs depending on the practical needs of their research settings. On the other hand, the advanced designs like multilevel and digital MM research designs are increasingly more often introduced in the JMMR and in the publications of Sage research methods series (e.g., Archibald et  al., 2019; Headley & Plano Clark, 2020; O’Halloran et  al., 2018), accompanied by innovative and iterative qualitatively driven MM designs (Creamer, 2018; Hesse-Biber, 2018). Their example will inspire a multitude of new designs to be constructed and introduced to the MM community. It is emphasized that the wicked problems that our societies face indispensably call for new theories, concepts and methods, especially in the evaluation-research tradition (Bamberger, 2016; Bamberger et al., 2016; Mertens et al., 2016). The transformative approach by Mertens (2013, 2015) has laid a solid foundation for this line of thinking and recently, some alternative lenses have been provided for an innovative integrative approach to complexity (Poth, 2018a, 2018b). MM approaches to address the wicked problems and grand challenges characteristic of the complexity of contemporary societies around the world are only in their early bulbing stage and will gain the endorsement of young generation of MM researchers. The empirical evidence shows that integration remains a challenge in actual research practice (Archibald et  al., 2019; Bazeley, 2018), but for successfully addressing the complexities and therefore bearing greater social impact, the potential of integration within the MM research designs needs to be wholly exploited. Being liberated by flexible frameworks for conceptualizing MM

72

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

designs and building on the iconic work done so far (e.g., Bazeley, 2013, 2018), new tools for integrating data and analyses in MM designs, as well as for visualizing integrated designs and the results of the analyses are to be exposed. Another important trend I envision is the alleviation of regional inequalities and the facilitation of culturally sensitive developments of MM designs, as fruitful discussions on the ways of constructing MM research designs are present in international editions like Kölner Zeitschrift für Soziologie und Sozialpsychologie (e.g., Schoonenboom & Johnson, 2017) or Studi sulla Formazione (e.g., Piccioli, 2019), and global discussions on various design options are welcomed outside these mainstream MM volumes (e.g., Kansteiner & Köning, 2020). Also, the regional chapters of MMIRA (Caribbean, Chinese, European, Japanese, Latin American and Oceanian) are helping the researchers of the regions to build a community and share expertise among them. Leaders of the chapters and experienced MM researchers with close backgrounds to the practitioners working professionally in the development and policy work of different regions are entitled to build stronger connections of the local members to the wider global networks of MM researchers benefitting, on one hand, by availed access to the latest international developments of MM research, and, on the other hand, being able to bring their perspectives to the global discourse. Creswell and Sinley (2017) emphasize that in order to facilitate the global expansion of MM, it is important to shape the discourse on the MM research to be more sensitive to diverse cultures that play a role in influencing methodological orientations and funding opportunities. This inevitably assumes an open-minded and responsible action of experienced MM scholars—for example, by providing extensive and competent training on MM research designs by mentoring international researchers interested in MM designs or helping to educate journal editors and reviewers of academic journals published around the world. The actions taken by MMIRA and current trends, scrutinized in this chapter, encourage me to envision the future where we, as an MM community, pay more and more attention to discourses being sensitive to diverse cultures, and provide malleable and contextually fitting ways for encouraging a greater extent of integration in MM designs. Having walked through the dominant pathways that I plotted in Figure 5.2, I hope you agree with me that the MM community has been enriched by a variety of influential lines of thought and taken each other’s ideas (sometimes also critique) as a basis for constructive development. The evidence shows that building bridges and having impelling

exchanges of the scholarship on the crossroads is how the further path for MM looks like. To conclude: for the wealthy future of MM, it should be recognized that the conceptual and methodological ideas on the ways of how to think about social phenomena and how to study them are developing rapidly: new philosophical schools of thought evolve and old ones are modified; new methodologies, methods and techniques are elaborated and experimented with; new influences bring novel focuses and viewpoints, etc. Being systematic but open-minded, flexible and inclusive in our ways of shaping and implementing MM research designs allows us to accommodate the natural development of the field and to validate our practices.

WHAT TO READ NEXT I encourage you to read Chapter 2 by Creswell and Plano Clark in this volume for a detailed overview of their continuous and fruitful work on MM design-typology development. It will be most helpful for the novice researchers and hopefully encourages us always to look for the newest texts from particular authors. Read Chapter 3 by J. Maxwell to widen your horizons in MM research by exploring the interactive model of design, which is one of the most often cited alternative approaches to MM typologies and find a number of exciting real-life examples of MM studies, which demonstrate that MM ideas have been and are often applied, even if there is no clear reference to the work of the MM community. Schoonenboom, J., & Johnson, R. B. (2017). How to construct a mixed methods research design. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(2), 107–131.

You will find a compact but rather comprehensive overview of different grounds the authors have proposed for describing and composing MM designs. They give special attention to design dimensions and provide discussion on typological versus interactive approaches to design. Tashakkori, A., Johnson R.B., & Teddlie, C. (2020). Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences (2nd ed.). Sage.

Out of the many MM textbooks, I suggest for the more advanced MM reader, this new edition as it collects the various ideas from the field and

DEVELOPMENTS IN MIXED METHODS DESIGNS

respectfully presents them, leaving room for the reader to take their stance.

REFERENCES Alise, M. A., & Teddlie, C. (2010). A continuation of the paradigm wars? Prevalence rates of methodological approaches across the social/behavioral sciences. Journal of Mixed Methods Research, 4(2), 103–126. https://doi.org/10.1177%2F1558 689809360805 Archibald, M., Radil, A., Zhang, X., & Hanson, W. (2015). Current mixed methods practices in qualitative research: A content analysis of leading journals. International Journal of Qualitative Methods, 5–33. https://dx.doi.org/10.1177/ 160940691501400205 Bamberger, M. (2016). The importance of a mixed methods approach for evaluating complexity. In M. Bamberger, J. Vaessen, & E. Raimondo (Eds.), Dealing with complexity in development evaluation (pp. 144–164). Sage. Bamberger, M., Vaessen, J., & Raimondo, E. (2016). Dealing with complexity in development evaluation. Sage. Bazeley, P. (2013). Qualitative data analysis: Practical strategies. Sage. Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage. Bergman, M. (2008). The straw men of the qualitative–quantitative divide and their influence on mixed methods research. In M. M. Bergman (Ed.), Advances in mixed methods research: Theories and applications (pp. 11–21.). Sage. Bergman, M. M. (2011). The politics, fashions, and conventions of research methods. Journal of Mixed Methods Research, 5(2), 99–102. https:// dx.doi.org/10.1177/1558689811408318 Brewer, J., & Hunter, A. (1989). Multimethod research: A synthesis of styles. Sage. Bryman, A. (1988). Quantity and quality in social research. Routledge. Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6(1), 97–113. https://doi.org/10.1177 %2F1468794106058877 Bryman, A. (2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1(1), 8–22. https://doi.org/10. 1177%2F2345678906290531 Cameron, R. (2009). A sequential mixed model research design: Design, analytical and display issues. International Journal of Multiple Research Approaches, 3(2), 140–152. https://researchportal. scu.edu.au/discovery/delivery/61SCU_INST:Research

73

Repository/1266897410002368?l#136745 0610002368 Cameron, R., & Miller, P. (2007). Mixed methods research: Phoenix of the paradigm wars. Managing our intellectual and social capital: Proceedings of the 21st ANZAM Conference. Caruth, G. (2013). Demystifying mixed methods research design: A review of the literature. International Journal of Education, 3(2), 112–122. http:// dx.doi.org/10.13054/mije.13.35.3.2 Creamer, E. (2018). An introduction to fully integrated mixed methods research. Sage. Creswell, J. (1995). Research design: Qualitative and quantitative approaches. Sage. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Sage. Creswell, J. W. (2009). Editorial: Mapping the field of mixed methods research. Journal of Mixed Methods Research, 3(2), 95–108. https://doi. org/10.1177/1558689808330883 Creswell, J. W. (2015a). A concise introduction to mixed methods research. Sage. Creswell, J. W. (2015b). Revisiting mixed methods and advancing scientific practices. In S. HesseBiber, & R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 61–71). Oxford University Press. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Sage. Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Sage. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Creswell, J. W., & Sinley, R. C. (2017). Developing a culturally-specific mixed methods approach to global research. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(2), 87–105. https://doi.org/10.1007/s11577-017-0453-2 de Vaus, D. A. (2004). Research design. In M. S. Lewis-Beck, A. E. Bryman, & T. Futing Liao (Eds.), The Sage encyclopedia of social science research methods (pp. 964–965). Sage. Doyle, L., Brady, A. & Byrne, G. (2016). An overview of mixed methods research – revisited. Journal of Research in Nursing, 21(8), 623–635. https://doi. org/10.1177/1744987116674257 Frels, R. K., & Onwuegbuzie, A. J. (2013). Administering quantitative instruments with qualitative interviews: A mixed research approach. Journal of Counseling & Development, 91(2), 184–194. http://dx.doi.org/10.1002/j.1556-6676.2013.00085.x Gage, N. (1989). The paradigm wars and their aftermath: A “historical” sketch of research and teaching since 1989. Educational Researcher, 18(7), 4–10. https://doi.org/10.3102/0013189X018007004

74

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Gambrel, L., & Butler VI, J. L. (2013). Mixed methods research in marriage and family therapy: A content analysis. Journal of Marital and Family Therapy, 39(2). 163–181. http://dx.doi.org/10.1111/j.17520606.2011.00260.x Gorard, S., & Taylor, C. (2004). Combining methods in educational and social research. Open University Press. Greene, J. C. (2007). Mixed methods in social inquiry. Jossey-Bass. Greene, J. C., & Caracelli, V. J. (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation, 1–94. http://eduq.info/ xmlui/handle/11515/15697 Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https:// doi.org/10.3102/01623737011003255 Guest, G. (2013). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7(2), 141–151. https:// doi.org/10.1177/1558689812461179 Hall, B., & Howard, K. (2008). A synergistic approach: Conducting mixed methods research with typological and systemic design considerations. Journal of Mixed Methods Research, 2(3), 248–269. https://doi.org/10.1177/1558689808314622 Hammersley, M. (1992). The paradigm wars: Reports from the front. British Journal of Sociology of Education, 13(1), 131–143. https://doi. org/10.1080/0142569920130110 Headley, M. G., & Plano Clark, V. L. (2020). Multilevel mixed methods research designs: Advancing a refined definition. Journal of Mixed Methods Research, 14(2), 145–163. https://doi.org/10.1177/ 1558689819844417 Hesse-Biber, S. (2018). Toward an understanding of a qualitatively driven mixed methods data collection and analysis: moving toward a theoretically centered mixed methods praxis. In The Sage handbook of qualitative data collection (pp. 545–563). Sage. Hesse-Biber, S.N., Rodriguez, D., & Frost, N. (2015). A qualitatively driven approach to multimethod and mixed methods research. In The Oxford handbook of multi and mixed methods research inquiry (pp. 3–20). Oxford University Press. https://doi. org/10.1093/oxfordhb/9780199933624.013.3 Ivankova, N. V. (2015). Mixed methods applications in action research: From methods to community action. Sage. Johnson, B., & Christensen, L. B. (2017). Educational research: Quantitative, qualitative, and mixed approaches. Sage. Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods

research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/1558689 806298224 Kansteiner, K., & König, S. (2020). The role(s) of qualitative content analysis in mixed methods research designs. Forum Qualitative Sozialforschung Forum: Qualitative Social Researc, 21(1), 1–22. www.qualitative-research.net/index. php/fqs/article/download/3412/4513?inline=1 Kroll, T. (2011). Designing mixed methods studies in health-related research with people with disabilities. International Journal of Multiple Research Approaches, 5(1), 64–75. https://doi.org/10.5172/ mra.2011.5.1.64 Lopez-Fernandez, O., & Molina-Azorin, J. F. (2011). The use of mixed methods research in the field of behavioural sciences. Quality & Quantity: International Journal of Methodology, 45(6), 1459–1472. https://doi.org/10.1007/s11135-011-9543-9 Lund, T. (2012). Combining qualitative and quantitative approaches: Some arguments for mixed methods research. Scandinavian Journal of Educational Research, 56(2), 155–165. https://doi.org/10.1080 /00313831.2011.568674 Mark, M. M. & Shotland, R. L. (1987). Alternative models for the use of multiple methods. In M. M. Mark, & R. L. Shotland (Eds.), Multiple methods in program evaluation: New directions for program evaluation (pp. 95–100). Jossey-Bass. Maxwell, J. A. (2013). Qualitative research design: An interactive approach. Sage. Maxwell, J. A. (2016). Expanding the history and range of mixed methods research. Journal of Mixed Methods Research, 10(1), 12–27. https:// doi.org/10.1177/1558689815571132 Maxwell, J. A., & Loomis, D. M. (2003). Mixed methods design: An alternative approach. In A. Tashakkori, & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 241–271). Sage. Maxwell, J. A., Chmiel, M., Rogers, S. (2015). Designing integration in mixed method and multimethod research. In S. Hesse-Biber, & R. B. Johnson (Eds.), Oxford handbook of multimethod and mixed methods research inquiry (pp. 223–239). Oxford University Press. Mayring, P., Huber, G.L., Gürtler, L., & Kiegelmann, M. (2007). Mixed methodology in psychological research. Brill. Mertens, D. M. (2013). What does a transformative lens bring to credible evidence in mixed methods evaluations? New Directions for Evaluation, 138, 27–35. https://doi.org/10.1002/ev.20055 Mertens, D. M. (2015). Mixed methods and wicked problems. Journal of Mixed Methods Research, 9(1), 3–6. https://doi.org/10.1177/15586898 14562944 Mertens, D.M. (2018). Mixed methods design in evaluation. Sage.

DEVELOPMENTS IN MIXED METHODS DESIGNS

Mertens, D. M., Bazeley, P., Bowleg, L., Fielding, N., Maxwell, J., Molina-Azorin, J. F., & Niglas, K. (2016). The future of mixed methods: A five year projection to 2020. MMIRA. Miller, S. I., & Fredericks, M. (2006). Mixed-methods and evaluation research: Trends and issues. Qualitative Health Research, 16(4), 567–579. https:// doi.org/10.1177/1049732305285691 Molina-Azorin, J. (2018). Mixed methods. In C. Cassell, A. L. Cunliffe, & G. Grandy The Sage handbook of qualitative business and management research methods (pp. 102–118). Sage. https:// dx.doi.org/10.4135/9781526430212.n7 Molina-Azorin, J. F., & Fetters, M. D. (2016). Mixed methods research prevalence studies: Field-specific studies on the state of the art of mixed methods research. Journal of Mixed Methods Research, 10(2), 123–128. https://doi.org/10.1177/15586898 16636707 Morgan, D. (2014). Integrating qualitative and quantitative methods. Sage. Morse, J., & Maddox, L. (2014). Analytic integration in qualitatively driven (qual) mixed and multiple methods designs. In The SAGE handbook of qualitative data analysis (pp. 524–539). Sage. Morse, J. M. (1991). Approaches to qualitative– quantitative methodological triangulation. Nursing Research, 40(2), 120–123. https://journals. lww.com/nursingresearchonline/Citation/1991/ 03000/Approaches_to_Qualitative_Quantitative. 14.aspx Morse, J. M. (2003). Principles of mixed methods and multi-method research design. In C. Teddlie, & A. Tashakkori (Eds.), Handbook of mixed methods in social and behavioral research (pp. 189–208). Sage. Morse, J. M. (2010). Simultaneous and sequential qualitative mixed method designs. Qualitative Inquiry, 16(6), 483–491. https://doi.org/10.1177/ 1077800410364741 Ngulube, P. (2010). Mapping mixed methods research in library and information science journals in SubSaharan Africa 2004–2008. International Information & Library Review, 42(4), 252–261. https://doi. org/10.1016/j.iilr.2010.10.005 Niglas, K. (1999). Quantitative and qualitative inquiry in educational research: Is there a paradigmatic difference between them? Paper given at ECER99. Education Line, 22–25. Niglas, K. (2004). The combined use of qualitative and quantitative methods in educational research. TPÜ Kirjastus. Niglas, K. (2009). How the novice researcher can make sense of mixed methods designs. International Journal of Multiple Research Approaches, 3(1), 34–46. https://doi.org/10.5172/mra.455.3.1.34 Niglas, K. (2010). The multidimensional model of research methodology: An integrated set of

75

continua. In SAGE handbook of mixed methods in social & behavioral research (pp. 215–236). Sage. Niglas, K. (2016). Expanding thinking through a kaleidoscopic look into the future: Implications of the Mixed Methods International Research Association’s task force report on the future of mixed methods. Journal of Mixed Methods Research, 10(3), 221–227. https://doi.org/10.1177/15586 89816649719 O’Halloran, K. L., Tan, S., Pham, D.-S., Bateman, J., & Vande Moere, A. (2018). A digital mixed methods research design: Integrating multimodal analysis with data mining and information visualization for big data analytics. Journal of Mixed Methods Research, 12(1), 11–30. https://doi.org/10.1177/ 1558689816651015 Octlund, U., Kidd, L., Wengström, Y., & Rowa-Dewar, N. (2011). Combining qualitative and quantitative research within mixed method research designs: a methodological review. International Journal of Nursing Studies, 48(3), 369–383. https://doi. org/10.1016/j.ijnurstu.2010.10.005 Parylo, O. (2012). Qualitative, quantitative, or mixed methods: An analysis of research design in articles on principal professional development (1998– 2008). International Journal of Multiple Research Approaches, 6(3), 297–313. https://doi.org/ 10.5172/mra.2012.6.3.297 Patton, M. (1980). Qualitative evaluation and research methods. Sage. Piccioli, M. (2019). Educational research and mixed methods. Research designs, application perspectives, and food for thought. Studi Sulla Formazione/Open Journal of Education, 22(2), 439–450. https://oajournals.fupress.net/index.php/sf/article/ download/10815/10797 Pinto, R. (2012). Mixed methods design. In N. Salkind (Eds.), Encyclopedia of research design (Vol. 1, pp. 813–818). Sage. Plano Clark, V., & Ivankova, N. (2016). How to use mixed methods research? Understanding the basic mixed methods designs. In Mixed methods research: A guide to the field (pp. 105–134). Sage. https://dx.doi.org/10.4135/9781483398341.n8 Plowright, D. (2011). Using mixed methods. Sage. Poth, C. (2018a). The curious case of complexity: Implications for mixed methods research practices. International Journal of Multiple Research Approaches, 10(1), 403–411. Poth, C. (2018b). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Rihoux, B., & Grimm, H. (Eds.). (2006). Innovative comparative methods for policy analysis: Beyond the quantitative-qualitative divide. Springer. Ross, A., & Onwuegbuzie, A. J. (2010). Mixed methods research design: A comparison of prevalence

76

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

in JRME and AERJ. International Journal of Multiple Research Approaches, 4(3), 233–245. https://doi. org/10.5172/mra.2010.4.3.233 Schoonenboom, J., & Johnson, R. B. (2017). How to construct a mixed methods research design. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(2), 107–131. https://doi.org/10.1007/ s11577-017-0454-1 Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Sage. Tashakkori, A., & Teddlie, C. (2003). Handbook of mixed methods in social and behavioural research. Sage. Tashakkori, A., & Teddlie, C. B. (2010). SAGE handbook of mixed methods in social & behavioral research (2nd ed.). Sage. Tashakkori, A., Johnson R.B., & Teddlie, C. (2020). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences (2nd ed.). Sage. Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In A. Tashakkori, & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 3–50). Sage. Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative

and qualitative approaches in the social and behavioral sciences. Sage. Truscott, D. M., Swars, S., Smith, S., Thornton-Reid, F., XZhao, Y., Dooley, C., Williams, B., Hart, L., & Matthews, M. (2010). A cross-disciplinary examination of the prevalence of mixed methods in educational research: 1995–2005. International Journal of Social Research Methodology, 13(4), 317–328. https://doi.org/10.1080/13645570903097950 van Meter, K. (2003). Interrelation between type of analysis and type of interpretation. Verlag Peter Lang. Vrkljan, B. H. (2009). Constructing a mixed methods design to explore the older driver—copilot relationship. Journal of Mixed Methods Research, 3(4), 371–385. https://doi.org/10.1177/15586 89809336843 Wisdom, J. P., Cavaleri, M. A., Onwuegbuzie, A. J., & Green, C. A. (2012). Methodological reporting in qualitative, quantitative, and mixed methods health services research articles. Health Services Research, 47(2), 721–745. https://doi.org/ 10.1111/j.1475-6773.2011.01344.x Zhou, Y., & Creswell, J.W. (2012). The use of mixed methods by Chinese scholars in East China: A case study. International Journal of Multiple Research Approaches, 6(1), 73–87. https://doi.org/10.5172/ mra.2012.6.1.73

DEVELOPMENTS IN MIXED METHODS DESIGNS

APPENDIX 1 The list of the educational texts on MM designs that were included in the sample of my overview. Cameron, R. (2009). A sequential mixed model research design: Design, analytical and display issues. International Journal of Multiple Research Approaches, 3(2), 140–152. https://doi.org/ 10.5172/mra.3.2.140 Caruth, G. D. (2013). Demystifying mixed methods research design: A review of the literature. Online Submission, 3(2), 112–122. https://eric.ed. gov/?id=ED544121 Creamer, E. (2018). An introduction to fully integrated mixed methods research. Sage. www.doi. org/10.4135/9781071802823 Curry, L., & Nunez-Smith, M. (2015). Mixed methods in health sciences research. Sage. www.doi. org/10.4135/9781483390659 DeCuir-Gunby, J., & Schutz, P. (2017). Developing a mixed methods proposal: A practical guide for beginning researchers. Sage. www.doi. org/10.4135/9781483399980 Doyle, L., Brady, A., & Byrne, G. (2019). An overview of mixed methods research – revisited. In SAGE mixed methods research. Sage. www.doi. org/10.4135/9781526498137 Edmonds, W., & Kennedy, T. (2017). An applied guide to research designs (2nd ed.). Sage. www. doi.org/10.4135/9781071802779 Piccioli, M. (2019). Educational research and mixed methods. Research designs, application perspectives, and food for thought. Studi Sulla Formazione/Open Journal of Education, 22(2), 439–450. https://doi.org/10.13128/ssf-10815 Plano Clark, V., & Ivankova, N. (2016). Mixed methods research: A guide to the field. Sage. www.doi. org/10.4135/9781483398341 Schoonenboom, J., & Johnson, R. B. (2017). How to construct a mixed methods research design. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(2), 107–131. https://doi.org/10.1007/ s11577-017-0454-1

APPENDIX 2 The list of references to the sources of MM design typologies that were mentioned in the educational texts included in my overview. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Sage.

77

Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Pearson. Creswell, J. W., & Plano Clark, V. (2007). Designing and conducting mixed methods research. Sage. Creswell, J. W., & Plano Clark, V. (2011). Designing and conducting mixed methods research (2nd ed.). Sage. Greene, J. C. (2007). Mixed methods in social inquiry. Jossey-Bass. Greene, J. C., & Caracelli, V.J. (Eds.). (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. Jossey-Bass. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https:// doi.org/10.3102/01623737011003255 Guest, G. (2013). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7(2), 141–151. https:// doi.org/10.1177/1558689812461179 Hesse-Biber, S. N. (2010). Mixed methods research: Merging theory with practice. Guilford Press. Hesse-Biber, S. N. (2010). Qualitative approaches to mixed methods practice. Qualitative Inquiry, 16(6), 455– 468. https://doi.org/10.1177/1077800410364611 Ivankova, N. V. (2015). Mixed methods applications in action research: From methods to community action. Sage. Johnson, B., & Christensen, L. B. (2017). Educational research: Quantitative, qualitative, and mixed approaches. Sage. Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26. https://doi.org/10.3102/00131 89X033007014 Leech, N. L., & Onwuegbuzie, A. J. (2009). A typology of mixed methods research designs. Quality & Quantity, 43(2), 265–275. https://doi.org/10.1007/ s11135-007-9105-3 Leech, N. L., & Onwuegbuzie, A. J. (2010). The journey: From where we started to where we hope to go [Editorial]. International Journal of Multiple Research Approaches, 4(1), 73–88. Maxwell, J. A., & Loomis, D. M. (2003). Mixed methods design: An alternative approach. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 241–271). Sage. Mertens, D. M. (2019). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods. (2nd ed.). Sage.

78

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. (2nd ed.). Sage. Morgan, D. L. (1998). Practical strategies for combining qualitative and quantitative methods: Applications to health research. Qualitative Health Research, 8(3), 362–376. https://doi.org/10.1177/ 104973239800800307 Morgan, D. L. (2014). v Integrating qualitative and quantitative methods: A pragmatic approach. Sage. Morse, J. M. (1991). Approaches to qualitativequantitative methodological triangulation. Nursing Research, 40(2), 120–123. https://journals. lww.com/nursingresearchonline/Citation/1991/ 03000/Approaches_to_Qualitative_Quantitative. 14.aspx Morse, J. M. (2003). Principles of mixed methods and multi-method research design. In C. Teddlie, & A. Tashakkori (Eds.), Handbook of mixed methods in social and behavioral research (pp. 189–208). Sage. Morse, J. M., & Niehaus, L. (2009). Mixed method design: Principles and procedures. Left Coast Press. Sandelowski, M. (2000). Combining qualitative and quantitative sampling, data collection, and

analysis techniques in mixed-method studies. Research in Nursing & Health, 23(3), 246–255. https://doi.org/10.1002/1098-240X(200006)23:3 3.0.CO;2-H Tashakkori, A., & Teddlie, C. B. (2010). SAGE handbook of mixed methods in social & behavioral research (2nd ed.). Sage. Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 3–50). Sage. Teddlie, C., & Tashakkori, A. (2009). Mixed methods research designs. In C. Teddlie & A. Tashakkori (Eds.), Foundations of mixed methods research: Integrating quantitative and qualitative approached in the social and behavioral sciences (pp. 137–167). Sage. Venkatesh, V., Brown, S., & Bala, H. (2013). Bridging the qualitative-quantitative divide: Guidelines for conducting mixed methods research in information systems. MIS Quarterly, 37(1), 21–54. www. jstor.org/stable/43825936

6 The Role of Methodological Paradigms for Dialogic Knowledge Production: Using a Conceptual Map of Discourse Development to Inform Mixed Methods Research Design Dawn Freshwater and Jane Cahill INTRODUCTION In this chapter, we focus on the important matter of knowledge production, with an emphasis on how the methodological paradigm of mixed methods research (MMR) has, and continues to, influence, inspire and diversify both the process of producing knowledge and, in turn, the knowledge that is produced. We argue that knowledge production is intimately connected with the variety of MMR designs we encounter as researchers. Our argument is influenced and substantiated by theories of paradigm development. As a result, we offer a theory of paradigm development, using a conceptual map of discourse development, which is experienced as a relational activity. Using a social justice in conjunction with a relational lens, we examine the notion of active participation in the development of discourses. This lens is especially relevant, as it relates to the lived experience of agency and power, thereby ensuring that discourse development is truly interactive and responsive, enabling innovation and empowering diversity in knowledge production. We aim to demonstrate that paradigm development is not an esoteric construct but an activity that has a bearing on the MMR design.

We return to and reflect on our previous work, as it relates to Section 1 of this text, inspiring design diversity and innovation. In this chapter, we will reflect on the process we went through, and that others have also gone through, in engaging with research, specifically writing, dialogue and the peer review of both our own work and that of other scholars. This process, which we term the dialogic process, underpins the role and function of the map of discourse development. We also note and develop the idea that dialogue is, in and of itself, a method of inspiring design diversity through relationship. Fundamentally, we view this chapter as an opportunity, within the Handbook, to cultivate a context in which multiple voices can be examined through a relational lens. We contextualize our approach to design diversity with reference to the other chapters in Section 1 of this Handbook which similarly examine evolutions and innovations in MMR designs. What we offer to this section on design is a relational perspective with the object of furthering design diversity. We draw on an ethic of reciprocity, incorporating complementary and competing perspectives abstracted from correspondence we had with Donna Mertens, John Creswell and other defining experts in

80

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

the field of MMR. We may include exchanges and responses from these experts in order to highlight the many and varied perspectives that co-exist, and inform both the process of knowledge production and the outputs of MMR. The conceptual map of discourse development presented within this chapter aims to solve a specific problem in the application of MMR: the discourse of MMR and the outcomes of the research itself. We suggest that MMR designs, along with all other research designs, can unwittingly perpetuate social injustice and inequity if conducted without self-critical awareness. It is crucial that the application of MMR does not entrench existing inequities, and, in doing so, compound social injustices. As Freshwater and Fisher (2014, p. 112) note: “Without an understanding of the political context, we identify a danger that MMR could be harnessed in ways that may inadvertently sustain and reinforce marginalization, including within the academy.” We make explicit the links between the conceptual map and MMR designs throughout this chapter, specifically within the section “Application of the conceptual map of discourse development to MMR”. We highlight in particular the deconstruction of previously espoused discourses, and those emerging, such that researchers may sharpen their focus on new ways of responding to existing problems.

BACKGROUND Research methodologies are constructed by diverse external and internal contextually driven influences. Accordingly, we identify two foci:

first, how the methodological paradigm of MMR continues to impact and be impacted by patterns of knowledge production. By knowledge production we are referring to the mechanisms by which knowledge is created and produced. We propose that the driving mechanism is participation in discourses (later explicated by the conceptual map). By discourses, we refer to the set of rules or assumptions for organizing and interpreting the subject matter of an academic discipline or field of study. And, as we later suggest, the development of these discourses is underpinned by responsiveness and relationality. Second, our analysis of the positioning of MMR uses a novel conceptual map of discourse development developed by the authors, which provides a framework to understand the epistemology for the generation of MMR research paradigm, research methods, and by extension research practices. Concerns about how MMR and outputs are affected by its underpinning methodological paradigms (quantitative, qualitative and MMR) have been well rehearsed within the existing literature. As such, we do not identify all the multiple authors and papers that have contributed to the debate here; rather, we select those contextual and defining pieces that directly underpin our argument. In addition, when referring to MMR, Morse (2006) voiced concerns about the emergence of confusing terminology resulting in a “mixed method design scramble” and pointed to largely quantitative methodologies incorporating qualitative research without proper consideration of the “principles of appropriate use” of qualitative data. Thus, as we employ a range of complex terminologies that are often taken for granted, misunderstood. or highly contested in the literature, we provide our key working definitions (see Box 6.1).

Box 6.1  Key definitions Term

Definition

Clinically representative research

Clinically representative research evaluates the effects of therapies as practised in real-world, routine settings. Clinically representative research encompasses highly diverse research designs and agendas, all of which purport to reflect real-world settings, and, in this supporting statement, covers effectiveness research and practice-based evidence. Set of rules or assumptions for organizing and interpreting the subject matter of an academic discipline or field of study. Effectiveness research aims to establish whether efficacious treatments are feasible and have measurable, beneficial effects across broad populations in real world settings. As defined in this supporting statement, effectiveness research in contrast to practice-based research imposes constraints found in research trial methodology such as randomization, treatment protocols and manuals. Efficacy research focuses on the measurable effects of specific interventions. In the clinical trial, one or more experimental treatments are compared with one or more control treatments. To maximize the likelihood of detecting treatment effects, factors that might obscure them are eliminated in efficacy studies as much as possible.

Discourse Effectiveness research

Efficacy research

THE ROLE OF METHODOLOGICAL PARADIGMS FOR DIALOGIC KNOWLEDGE PRODUCTION

Epistemology

Evidence-based practice

Mixed methods research Paradigm Practice-based evidence

Qualitative paradigm

Quantitative paradigm

Systematic review

81

Branch of philosophy concerned with the nature and origin of knowledge. As the study of knowledge, epistemology is concerned with the following questions: What are the necessary and sufficient conditions of knowledge? What are its sources? What is its structure and what are its limits? Evidence-based practice promotes the collection, interpretation and integration of valid, important and applicable patient-reported, clinician-observed and research-derived evidence. The best available evidence, moderated by patient circumstances and preferences, is applied to improve the quality of clinical judgements and facilitate cost-effective care. Mixed methods research is a methodology for conducting research that involves collecting, analyzing and integrating (or mixing) quantitative and qualitative research (and data) in a single study or a longitudinal programme of enquiry. Ways of looking at the world that define both the problems to be addressed and the range of admissible evidence that may be used to effect solutions. Practice-based evidence promotes the collection of evidence derived from routine settings and measures what normally happens in practice settings. Practice-based evidence is driven by the aim of showing that procedures work and are effective in improving the quality of care in real-life practice settings. Qualitative methodology has been frequently defined in terms of opposition to quantitative methodology and as a field is much more fragmented. To summarize, qualitative research relies largely on words, narratives and clinical judgement. Qualitative research has been viewed as particularly suited to investigating the personal meanings that people attach to their experiences and for investigating the processes (macro, mid and micro levels) of the phenomenon or phenomena under investigation. This methodology is reliant on numbers and statistical analyses. Quantitative methods are used in various ways to advance knowledge about the processes and outcomes of phenomena. Quantitative methods can be helpful because they allow us to study the complex interplay of relations between variables and to identify overarching patterns derived from numbers or statistics. Once a pattern is identified, it is then possible to hone in on particular aspects of the phenomena in question to conduct more in-depth qualitative research. Systematic reviews are designed to provide a comprehensive accounting of all evidence relating to a particular question or problem area. Studies are located through methods such as electronic database searching, and the review follows a protocol to determine which studies have the necessary level of quality and are thus eligible for inclusion in the review of the available literature.

Source: Author created.

To advance new understandings of MMR, we particularly focus on the discourses that surround, sculpt, and propel research and research methods. As part of our analysis, we present a conceptual map of discourse development which we suggest can be used as a heuristic device to understand and critically reflect on the development of research discourses such as MMR. We situate critical reflection as central to our analysis throughout. We begin by critically examining and deconstructing the conceptual foundation of paradigms, which has specific implications for the framing of MMR, a discourse that has huge purchase in current healthcare policy and practices.

What Is a Paradigm? There has been long-standing substantial variation in how people understand the construct of

a “paradigm”. Thomas Kuhn’s (1996) seminal definition referred to a set of practices that characterize a scientific discipline at any particular period of time. This definition affords some degree of slippage. One standpoint, exemplified by Mertens (2007, 2010) more than a decade ago, contests that paradigms must comprise sets of philosophical assumptions with regard to methodology, epistemology, ontology and axiology. In this model, methodological assumptions can determine a choice of methods: quantitative, qualitative or mixed in several paradigms—most commonly, the pragmatic and transformative paradigms. The key epistemological premise is that the paradigm is a higher order construct that “sires” or “begets” choices in methods. There is an alternative school of thought that permits paradigms to be methodological in their foundation. As the debate around MMR developed, so the scholarship questioning the implied

82

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

and explicit paradigmatic stance of MMR has ballooned. For example, some time ago now, Denscombe (2008) and Johnson and Onwuegbuzie (2004) dubbed the MMR approach the “third paradigm” for social research in its synthesis of quantitative and qualitative methodologies. These same scholars also consider the philosophical and epistemological underpinnings of paradigms from a pragmatist standpoint. Denscombe (2008) noted that the communities of practice research paradigm is consistent with the pragmatist underpinnings of the MMR approach. By pragmatism we are referring to its application in MMR to denote a problem-solving mindset whereby the “best” research methods to answer the research question are adopted. We firmly contend that when we are writing about methodological paradigms, we are not simply referring to choices of methods or methodological procedures, but denoting an epistemological construct that has specific impacts on how we position and understand MMR. We fully acknowledge that these two quite distinct understandings of “mixed methods” are used interchangeably and often conflated, leading to conceptual mayhem. Although both of us as authors have participated in considerable debate on this difference in understanding, most notably with Donna Mertens (as evidenced by our correspondence and editorials in the Journal of Mixed Methods Research), we also recognize that when conceptual understandings are used interchangeably on a repeated basis, boundaries become porous. As a result, the distinctions so keenly felt by ourselves are not experienced as such by all members of the MMR community, with subsequent implications for the agency and power of those involved in the research, both as participants and researchers, and indeed the social justice of the research question and its outcomes. As Holloway (2011) observed, the use of the term “paradigm” has become problematic through being freely used but not interrogated for meaning. MolinaAzorin and Fetters (2020) provide a compelling summary of the debate on paradigm in MMR. Biddle and Schafft (2014) have called for greater epistemological engagement that is inherent in pragmatism specifically in regard to its axiology. They suggest that the transformative paradigm would answer to this gap. This argument sets the tone for the debate to follow which centres around merits of different paradigms, the use of multiple paradigms (Hatchcoat & Meixner 2017; Johnson, 2017; Shannon-Baker 2016) or sees the introduction of new paradigms and analytical frameworks (Cronenberg, 2020; Ghiara, 2020; Schoonenboom, 2019). We certainly concede this issue in our own inquiry, in nailing our epistemological colours

to the mast, we contend that a “methodological approach” can form the basis of a paradigm, which can indeed be conceptualized as having its own epistemological, ontological and axiological assumptions. We would like to disabuse the reader of any notion that we are suggesting that sets of methods—whether quantitative, qualitative or mixed—are paradigms. In keeping with the notion of paradigm refinement and development outlined later in this chapter, there is scope for diverse conceptualizations ranging from higher order philosophical paradigms that beget choices in methods, and paradigms that can be methodological in their foundation. We would highlight that in the latter definition, we conceptualize methodologies themselves as not only choices of methods, but as epistemological standpoints with their own conceptual and philosophical underpinnings. In the next section, we outline and define some conceptual issues relating to research practice, which are themselves contingent on how paradigms are conceptualized.

What Is Research Practice? Definitions of research practice are fluid and contingent. In this context, we define research practices as the operationalization and implementation of ideologies inherent in research methods and designs. Espoused theories, held dearly, flex and change as they become theories in action (Freshwater, 2008). Discourses around research methods perpetuate research practices, which in turn validate and support the dominant discourses associated with research methodology. Thus, discourse is both subject (perpetuating) and object (perpetuated), in this cycle, which ensures that dominant discourses retain their privileged position. Unless discourses are informed by and are responsive to variation in research practices, they remain largely idealistic and theoretical. Our conceptual map allows us to more closely reflect on the processes whereby research methods are constructed by and feed back into the discourse. Research publication is one example of a research practice that illustrates well its pivotal role of supporting and perpetuating discourses surrounding research methodologies and their application to academic and political knowledge production.

Social justice, agency, power, and politics

Contemporary research enquiry is directed towards and favors interdisciplinarity, along with the application of MMR. This trend is accompanied by calls for a sensible and pragmatic approach

THE ROLE OF METHODOLOGICAL PARADIGMS FOR DIALOGIC KNOWLEDGE PRODUCTION

to knowledge production, which provides a way of overcoming traditional disciplinary, methodological and paradigm wars associated with factionalism and circular conflict, and, importantly, has greater measurable impact and translation to end user outcomes. In this sense, research application is a key driver for researchers–practitioners, funding agencies and academics alike. Even so, it is important to consider the risks that might arise from the application of MMR, when divorced from reflection on the political dimensions of knowledge production. We raise this, as it is our position that without an understanding of the political context of knowledge production, there is a danger that the process and outcomes of the MMR could be harnessed in ways that inadvertently reinforce marginalization of particular groups (Freshwater & Fisher, 2014). To reiterate, it cannot automatically be assumed that MMR contributes to the production of contextualized knowledge with benefits to multiple sections of society. One needs to question whether it is not merely serving the interests of the privileged voices of the academics and theoreticians who are determining the methodology. Similarly, there is a risk that when the drive for integrated and applied findings becomes all-consuming, peripheral or marginal perspectives can be overlooked. Much has been written about the relations of power in the definition and construction of problems, but such processes are not always subject to critical evaluation, or critical reflection. Social justice research attempts to interrogate some of the underlying, often unconscious or unquestioned constructs that are hidden in the crevices of the research approach, methodology, and processes, attending carefully to the principles of equity, access, and participation. As an approach, social justice research represents a multifaceted approach in which investigators strive to simultaneously promote human development and the common good through addressing challenges related to both individual and distributive justice. Such research places emphasis on empowerment of the individual, as well as the active confrontation of injustice and inequality in society as they affect research participants. Mertens (see also Chapter 4, this volume) has written about this extensively and has provided a number of case studies to illustrate the complexity of social justice outcomes as part of her transformative research theory (see, for example, Mertens, 2007). She noted that “power is an issue that must be addressed at each stage of the research process” (Mertens, 2007, p. 213). We agree, and, as Freshwater has noted elsewhere, that in this regard every interaction is a possible intervention and should be thought about and treated

83

as such. Such interactions include the deliberative and intentional discussion of any subsequent positive and negative impact of the research, short and long term, intended and/or unintended, before, during and after the research is undertaken. Mertens (2007) went on to say that “the use of a single method to determine the need for social change … can yield misleading results” (p. 214). The point she makes is clear: good MMR is needed because lives are at stake, and most often those lives belong to the already disenfranchised. As a tool, to aid in the “surfacing” of previously unquestioned or unconscious processes, we suggest that the conceptual map of discourse development could be helpful. The map can be used explicitly as a heuristic aid to interrogate the knowledge production and the power relations therein.

CONCEPTUAL MAP OF DISCOURSE DEVELOPMENT The conceptual map was partly derived from a review of research articles published in the Journal of Psychiatric and Mental Health Nursing. This was an exercise conducted in support of the Journal, but a by-product became the authors’ theories of knowledge production (Freshwater & Cahill, 2009). Note that the theoretical, or conceptual structure, of the map was primarily derived from the second author’s modelling of the therapeutic relationship (Cahill et  al., 2008; Hardy et al., 2007) which lists three key developmental processes as necessary for the sustainability of an effective therapeutic relationship—establishing a relationship, developing a relationship, and maintaining a relationship. By way of introducing the relational basis of discourse development, we will set out what we believe to be the structural premise of paradigm formation. Freshwater and Rolfe (2004, p. 58) cite Thomas Kuhn’s definition of a paradigm as “ways of looking at the world that define both the problems that can legitimately be addressed and the range of admissible evidence that may bear on their solutions”. The authors then go on to define a discourse as a “set of rules” or “assumptions for organizing and interpreting the subject matter of an academic discipline or field of study” (p. 135). We view discourses as underpinning paradigms, so in our theoretical model of understanding, the paradigm is the explanatory framework/structure and a discourse is the “set of rules” and “assumptions for organizing and interpreting subject matter”. So, the enactment of the discourse (which

84

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

is a practice) “builds” the paradigm. We contend that these sets of rules and assumptions are constantly open to dynamic processes generated when the reader or audience responds to the discourse; as such, this process is inherently relational. For these reasons, we conceptualize discourse and ultimately paradigm development in relational and dynamic terms, and draw parallels with dynamic processes observed in the formation of a therapeutic relationship. The use of the map of the therapeutic relationship to inform our conceptual map of discourse development is far from arbitrary; we contest that discourses are generated in dynamic processes and that are they iterative and responsive to contextual factors (which we capture in the conceptual model). We would further highlight that the context of this map is in healthcare practices, which are in essence relational. If we hold that paradigm development is relational, this proposed conceptual framework also enables the “consumer” of a research paradigm to assume a more active stance to position themselves in relation to the discourse and influence its developmental trajectory. The idea of active participation in discourse development is not simply theoretical posturing, but an expression of the lived experience of agency and power, and a potential strategy for preventing hegemony in healthcare and educational practice. Therein, addressing issues related to social injustice. What follows is an overview of the map and a description of its components. We conclude with

some examples of research, scholarship and practice that illustrate the configuration of the map in relation to the MMR paradigm and the impact on research.

Overview of the Map We suggest that the conceptual map (see Figure 6.1 below) can be used as a heuristic device to understand the following: research processes, research methodologies and their reproduction; the formation of research paradigms and how stories are created; and knowledge production, its perpetuation and maintenance. These considerations provide a statement on knowledge generation, knowledge transfer and its impact on academic disciplines. Furthermore, we contend that the conceptual map of discourse development addressed a fundamental concern in both the critique and application of MMR—that is, the importance of developing alternate ways of responding to existing real-world problems, outside the dominant discourse of MMR. The failure to do so, will, we argue, commit researchers to entrenched ways of responding to those that are marginalized, experience social injustice, and are subject to inequitable decision-making. In providing a schematic overview, we begin our description from the left—the section of the map concerned with “creation of a discourse”.

Figure 6.1  Conceptual map of discourse development Source: Author created.

THE ROLE OF METHODOLOGICAL PARADIGMS FOR DIALOGIC KNOWLEDGE PRODUCTION

Four key developmental processes which have been identified as being necessary for facilitation of a discourse are: 1 2 3 4

Establishing a discourse. Maintaining or perpetuating a discourse. Developing a discourse. Deconstructing a discourse.

This last developmental process is in addition to the processes outlined in Cahill et al. (2008) and Hardy et al. (2007),and is presented as a process directly resulting from development of discourse rather than as a discrete phase (Freshwater, 2007a, 2007b). We also highlight that in contrast to the map of the therapeutic relationship (Figure 6.2), we position the “developing” after the “maintaining” phase; this seemed most appropriate to our model in that we view subsequent development or deconstruction of a discourse as succeeding a period of stability or maintenance. We include the original map of the therapeutic relationship to indicate how the conceptual map of discourse development has been grounded on psychotherapeutic principles. The “learning to be part of a discourse” process in the central part of the map is cyclical, regenerative. and multidirectional. The tangible outputs of this are publications, which in turn impact on all processes of discourse development. Key contextual factors in Figure 6.1 are grouped into external and internal (researcher and consumer) factors which play a significant part to determine the nature of the learning process, which in turn impacts the developmental stages of discourse

Figure 6.2  Map of therapeutic relationship Source: Author created.

85

development. We acknowledge that this is a somewhat unidirectional description, but the block arrows signify the cyclical nature of research practice with discourses feeding back into academic scholarship and impacting on contextual factors. In the sections that follow, we focus on explication of the four key developmental processes with reference to MMR.

Application of the Conceptual Map of Discourse Development to MMR We can attribute the development of MMR to contextual factors such as consumers (in this case practitioners or researchers) seeking meaningful research that applies to a variety of methodological orientations that are not necessarily aligned purely with quantitative or qualitative paradigms. Indeed, it has been proposed that MMR grew from the “paradigm wars” where, after the ascendance of quantitative methodologies between the 1950s and 1970s, and qualitative methodologies from the 1970s to the 1990s, it emerged as a bridge between the two (Denscombe, 2008) and has since been constructed by its proponents as the third paradigm, a “separate methodological orientation with its own worldview, vocabulary and techniques” (Tashakkori & Teddlie, 2003, p. 112), and which has both object (produced by the paradigm wars) and subject (impacting on how MMR is positioned) roles. Thus, integrating methods has a long history, as reported by many key MMR authors. So, while it is true that the

86

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

institutionalization of MMR as a field started after the paradigm wars, MMR have a prior history that considerably predates the time when it became formalized as a research field; however, we would nevertheless suggest that the “paradigm” wars still provided a crystallizing moment for MMR and facilitated a distinct shift in focus. In determining the creation of the underpinning discourse, we once again consider separately the establishing, maintaining and developing phases. The establishment of the discourse has, in part, been activated by seminal publications that promote the distinctive nature of the paradigm and its core ideas and practices (Creswell, 2003; Creswell & Plano Clark, 2007; Denzin & Lincoln, 2011; Tashakkori & Teddlie, 1998, 2003) by high-quality publications and by a dedicated readership, as indicated, for example, by the statistics in the Journal of Mixed Methods Research and indeed in this Handbook. In the maintenance phase, a way to actively perpetuate the discourse has been to ensure that research outputs are monitored through quality control methods so as to ensure highimpact publications that have the potential to attract funding streams. To this end, it has been essential to include checks on quality control in terms of publishability, and on the specific contribution each publication makes to the field (Creswell & Tashakkori, 2007, 2008; Mertens, 2011). Examples of seminal research publications which have focused on methodological improvements and advances in the field are to be found in articles on paradigmatic formulations and innovative thinking about designs. In relation to the former, we would refer readers to Morgan’s (2007) and Denscombe’s (2008) explication of the community-of-scholars idea. This line of thought is pivotal for the development of the underpinning discourse in that it accommodates the fragmentations and inconsistencies previously eschewed by researchers advocating integration (Bryman 2007, 2008) in the MMR approach (Creswell, 2011). In this chapter, Creswell notes 11 controversies in MMR, a discussion which has been prominent in the qualitative community in the USA and is part of the process of deconstructing, challenging, and ultimately strengthening the emerging field. The development and subsequent deconstruction phases are the most pressing for MMR researchers and practitioners in that they will directly impact the future in terms of how the discourse can be advanced. Creswell and Plano Clark (2010) have termed this as the “reflective” phase. One of the ways a discourse can be advanced is

paradoxically through its deconstruction and attendant dissonance. We suggest that this development phase is the one in which most significant work takes place in terms of the reflection, questioning, and the interrogation of previously held assumptions. Given that social justice is concerned with surfacing previously unconscious or unquestioned assumptions, we suggest that these phases present the most dynamic space in which social justice matters can best be served. There are several ways that we can focus on fractures and anomalies during the deconstruction phase in any given discourse. First, there is the approach of accommodating variations, inconsistencies and fragmentations in the discourse to strengthen the paradigm. For example, Bergman (2008) and Denscombe (2008) use the “communities of practice model” to formulate a model of paradigm development based on smaller communities of practice. According to this model, research practitioners use such ideas as shared understanding, shared identity, practice-driven approach to research problems, informal networks and groupings. Above all, a flexible approach to inquiry that incorporates the inconsistencies and fragmentations in discourses underpinning MMR offers a responsive approach to any given research problem. There are multiple formulations of the anomalies in MMR; for a deeper understanding of such critiques, we recommend authors such as Joseph Maxwell (see also Chapter 3, this volume), Max Bergman, Stephen Gorard, Lynne Giddings or Uwe Flick. Another way of addressing anomalies and fractures in discourses surrounding MMR is offered by Freshwater’s (2007a, 2007b) postmodern critique. Here, the emphasis is not so much on the content of the MMR discourse, as in the reading and writing practices that are perpetuating the discourse. Freshwater deals with the “consumers”——the health and social care researchers—who in their eagerness to become part of the academic discourse have displayed an uncritical and unquestioning stance in their reading of MMR, believing it to be a panacea for the solution of the unsolvable, wicked problems, some of which are explicitly related to societal inequities. While interpreting the discourse as one that integrates and fuses dialectical and opposing paradigms has been employed to overcome uncomfortable tensions, this has led to flatness in the quest for unity across methodological approaches, a unity promoted as enhancing validity. There has been a trend for pinning down internal and competing components to present a coherent and comprehensive map of the area, a practice

THE ROLE OF METHODOLOGICAL PARADIGMS FOR DIALOGIC KNOWLEDGE PRODUCTION

which directly bears on Freshwater’s critique. Creswell notes this tension in his 2009 editorial on mapping the field: while recognizing that a mapping exercise can be interpreted as an attempt to fix the field and provide a template to which new components must be assimilate, Creswell also argues that the map is simply the beginning of a conversation rather than an attempt to impose determinacy (see also Chapter 2, this volume).

IMPLICATIONS FOR MIXED METHODS RESEARCH DESIGN PRACTICE In reflecting on our motivation for developing the conceptual map, we recognize that it was partly down to an attempt to understand the complex and multi-layered way the MMR paradigm continues to influence the direction of healthcare research. However, we also wish to highlight that the conceptual map is not restricted to healthcare research but can apply to other disciplines and contexts and is, as such, transdisciplinary. Healthcare research practice is simply employed as an exemplar. As described earlier, this map’s genesis was its predecessor: the conceptual map of the therapeutic relationship (Cahill et al., 2008) developed within the disciplines of psychology, psychotherapy, counselling and medicine. So first, the map presented here is by no means restricted to the healthcare context but is naturally applicable to disciplines which are predicated on the study of interactions with others. Second, its wide applicability suggests that our conceptual map has offered a meta-perspective, pointing to generic factors of discourse development which in turn underpin research paradigms. What the conceptual map offers is transferability across discipline boundaries which in turn opens up the map to critique and advancement in the service of design diversity which brings us back to the focus of Section 1 in which our work is situated. However, we would like to acknowledge some danger inherent in the approach of offering an overarching meta-perspective that does to some degree present as a meta-narrative. We have not only described how discourses underpin the production and practice of methodologies but also have presented a narrative about the development of discourses themselves, a narrative which in a sense becomes self-perpetuating. We suggest that the self-perpetuating nature of discourse, if the production of them goes unchecked, can result in lack of agency, and indeed responsibility which may become a breeding ground of

87

social inequities. As such, we would urge readers and future researchers to apply the map to their own discipline and context, and in the spirit of deconstruction, to extend, critique and indeed to contest the map. What the map does offer is an inroad into how consumers—healthcare researchers, practitioners and policy-makers—can take an active stance in how a given research paradigm might develop in the future. Freshwater (2007a, 2007b) pointed to the drawbacks of consumers adopting an uncritical reading of MMR which results in a bland landscape; fusion and integration are privileged over uncertainty and paradox. The converse is that by harnessing critical abilities in becoming part of an academic discourse, we, as members of the healthcare community, can offer alternative readings of any given research paradigm that celebrate rather than occlude tensions. In this sense, rather than being simply written into the paradigm and hence having our research methods predetermined, we can make the decision to live with and exploit tensions, potentially effecting paradigm shifts, and by so doing surfacing issues of power and injustice. We would also like to highlight the ways that our conceptual map has an impact not only on the paradigm of MMR, but also on the debate concerning what constitutes paradigms themselves. Based on our own knowledge of paradigm development, we would contend that “reading” and “writing” on the nature of paradigms and their conceptual ingredients necessarily involve disparate viewpoints in the academic healthcare community. We suggest that readers and writers respond to and interact with research outputs, of which ours is an example, in a variety of unpredictable ways. These understandings, or misunderstandings as they might be termed, then lead to iterations that contribute to the development and ultimately deconstruction of discourses. We suggest that in modelling our map of discourse development (Figure 6.1) on the dyadic therapist–client relationship in psychotherapy, we are arguing for an epistemology of MMR knowledge that is grounded in responsiveness and symbiosis. We can view this as an extension or variation of the communities of practice basis of paradigm development (Denscombe, 2008). Taking on board the idea that research paradigms are based on smaller communities with shared identities, informal networks, and groupings and relational practices, we drill down even further to an explanation of paradigm formation in modelling it at the micro level of the dyadic relationship. This relational basis of discourse development is fluid, contingent and dynamic.

88

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

LOOKING AHEAD • The conceptual map of discourse development should be used to provide a framework to understand and critically reflect on the epistemology for the generation of research paradigms and research methods, and, by extension, research practices. • The conceptual map should be used to cultivate an awareness in practitioners, researchers and policy-makers of how discourses surrounding research evidence and research practices are generated; this, in turn, may facilitate critical reflection on how certain practices assume dominance, potentially leading to hegemony in nursing research, practice, and scholarship. • We suggest that the conceptual map should be deployed in providing an inroad into how consumers—that is, researchers, practitioners and policy-makers—can take an active stance in how a given research paradigm might develop in the future. So, as consumers, rather than being simply written into the paradigm and hence having research methods predetermined, we can make the decision to live with and exploit tensions, and effectively rewrite ourselves into the paradigm so as to potentially effect paradigm shifts. • Our studies, over time, have generated a conceptual map that outlines the generic factors of discourse development, which in turn underpin research paradigms and the subsequent designs. Modelling our map of discourse development on the dyadic client relationship in psychotherapy, we offer an epistemology of knowledge production that is grounded in relationality, responsiveness and symbiosis. • We contest that research methodologies are constructed by discourses that are themselves dynamic and relational. This proposed theory thus offers consumers of research a model of how to actively influence production and development of methodologies. • This is important, as matters of social justice are best served in the development and deconstruction phases that facilitate “surfacing” of previously unquestioned and unconscious assumptions. • We recommend that the conceptual map should be used and in future work be refined according to differing contexts, as a new method in the research community to cultivate an awareness in healthcare practitioners, researchers and policy-

makers of how discourses relating to research evidence and research practices are produced and perpetuated, and inform the research design. • Engendering active and critical reflection on the generation of these practices and the ways they can be deployed, in healthcare research, practice and scholarship is, we suggest, an integral part of advancing healthcare knowledge and practice. Importantly, we propose that critical reflection on discourse development in MMR will better enable the research design, the questions being posed and the findings generated to lend themselves to a range of recipients of care, decisions and practices informed by a knowledge production schema that holds social justice, agency and equity as central tenets. • We have argued that research methodologies are constructed by external and internal contextually driven influences and the concerns about how healthcare and related research has been positioned by the methodological paradigm of MMR has been well rehearsed in the literature. We note that there is substantial variation in how people understand the construct of ‘paradigm’; this research critically reflects on the implications of such variation, and indeed discrepancy, for the research community. As a final postscript, we would urge that MMR, its application, does not inadvertently entrench existing inequities and, in doing so, compound social injustices. Although we have presented our conceptual map as a reflexive tool in order to mitigate this danger, we do, of course, acknowledge the danger of the map becoming a meta-narrative and monolithic in its application. This is where we would suggest a collective and ongoing responsibility among all consumers and users of MMR.

WHAT TO READ NEXT Freshwater, D., & Fisher, P. (2014). (Con)Fusing commerce and science: MMR and the production of contextualised knowledge. Journal Mixed Methods Research, 8(2), 111–114. https://doi.org/10.1 177%2F1558689814526804

This editorial offers an essential introduction to knowledge production, its pursuit and pitfalls, and makes the case that mixed methods research could be harnessed in ways that may inadvertently sustain and reinforce marginalization, including within the academy.

THE ROLE OF METHODOLOGICAL PARADIGMS FOR DIALOGIC KNOWLEDGE PRODUCTION

Johnstone, P. L. (2004). Mixed-methods, mixed methodology: health services research in practice. Qualitative Health Research, 14(2), 259–271. doi:10.1177/1049732303260610

The article is an illustrative example describing a researcher’s decisions to adopt mixed methods research, and offers a helpful conceptual model of the research process. Chapter 4 in this volume by Donna Mertens. Mertens describes the potential of mixed methods research to contribute to increased justice and advances the use of a transformative lens incorporating ethical responsibility to design studies that are culturally responsive.

REFERENCES Bergman, M. M. (2008). Advances in mixed methods research. Sage. Biddle, C., & Schafft, K. A. (2014). Axiology and anomaly in the practice of mixed methods work: Pragmatism, valuation, and the transformative paradigm. Journal of Mixed Methods Research, 9(4), 320–334. https://doi.org/10.1177 %2F1558689814533157 Bryman, A. (2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research 1(1), 8–22. https://doi.org/10. 1177%2F2345678906290531 Bryman, A. (2008). Social Research Methods (3rd ed.). Oxford University Press. Cahill, J., Barkham, M., Hardy, G., Gilbody, S., Richards, D., Bower, P., et al. (2008). A review and critical appraisal of measures of therapist patient interactions in mental health settings. Health Technology Assessment, 12(4), 1–68. Retrieved from: www.hta.ac.uk/execsumm/summ1224.shtml Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Sage. Creswell, J. W. (2011). Controversies in mixed methods. In N. K. Denzin & Y. S. Lincoln (Eds.), SAGE handbook of qualitative research (4th ed.) (pp.269–283). Sage. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Sage. Creswell, J. W., & Plano Clark, V. L. (2010). Designing and conducting mixed methods research (2nd ed.). Sage. Creswell, J. W., & Tashakkori, A. (2007). Developing publishable mixed methods manuscripts. Journal of Mixed Methods Research, 1(2), 107–111. https://doi.org/10.1177%2F1558689806298644

89

Creswell, J., & Tashakkori, A. (2008). How do research manuscripts contribute to the literature on mixed methods? Journal of Mixed Methods Research, 2(3), 115– 120. https://doi.org/10.1177%2F1558689808315361 Cronenberg, S. (2020). Paradigm parley: A framework for the dialectic stance. Journal of Mixed Methods Research, 14(1), 26–46. https://doi.org/ 10.1177%2F1558689818777925 Denscombe, M. (2008). Communities of practice: A research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2(3), 270–283. https://doi.org/10.1177%2F1558689808316807 Denzin, N. K., & Lincoln, Y. S. (2011). The SAGE handbook of qualitative research (4th ed.). Sage. Freshwater, D. (2007a). Discourse, responsible research and positioning the subject. Journal of Psychiatric & Mental Health Nursing, 14(2), 111–112. https://doi. org/10.1111/j.1365-2850.2007.01089.x Freshwater, D. (2007b). Reading mixed methods research: Contexts for criticism. Journal of Mixed Methods Research, 1(2), 134–145. https://doi.org/ 10.1177%2F1558689806298578 Freshwater, D. (2008). Reflective practice: The state of the art. In D. Freshwater, B. J. Taylor, & G. Sherwood (Eds.) International textbook of reflective practice in nursing (pp. 118). Wiley-Blackwell. Freshwater, D., & Cahill, J. (2009, July). Practice of publishing research: A conceptual map. Paper presented at the International Mixed Methods Conference, Harrogate, United Kingdom. Freshwater, D., & Fisher, P. (2014). (Con)Fusing commerce and science: MMR and the production of contextualised knowledge. Journal Mixed Methods Research, 8(2), 111–114. https://doi.org/10. 1177%2F1558689814526804 Freshwater, D. and Rolfe, G. (2004). Everything and nothing: Deconstructing evidence based practice. London: Routledge/ Taylor and Francis. Ghiara, V. (2020). Disambiguating the role of paradigms in mixed methods research. Journal of Mixed Methods Research, 14(1), 11–25. https:// doi.org/10.1177%2F1558689818819928 Hardy, G., Cahill, J., & Barkham, M. (Eds.) (2007). Active ingredients of the therapeutic relationship that promote client change: A research perspective. Routledge/Taylor & Francis. Holloway, I. (2011). Being a qualitative researcher. Qualitative Health Research, 7, 968–975. https:// doi.org/10.1177%2F1049732310395607 Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14– 26. https://doi.org/10.3102%2F00131 89X033007014 Johnson, R. B. (2017). Dialectical pluralism: A metaparadigm whose time has come. Journal of Mixed Methods Research, 11(2), 156–173. https://doi. org/10.1177%2F1558689815607692

90

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Kuhn, T. S. (1996). The structure of scientific revolutions (2nd ed.). University of Chicago Press. Mertens, D. M. (2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research, 1, 212–225. https://doi. org/10.1177%2F1558689807302811 Mertens, D. M. (2010). Transformative mixed methods research. Qualitative Inquiry, 16(6), 469–474. https://doi.org/10.1177%2F1077800410364612 Mertens, D. M. (2011). Publishing mixed methods research. Journal of Mixed Methods Research, 5(1), 3–6. https://doi.org/10.1177%2F15586898 10390217 Molina-Azorin, J. F., & Fetters, M. D. (2020). Virtual special issue on “Paradigms in mixed methods research”. Journal of Mixed Methods Research, 14(1), 6–10. https://doi.org/10.1177%2F1558689819894741 Morgan, D. (2007). Paradigms lost and pragmatism regained: Methodological implications of

combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1). https:// doi.org/10.1177/2345678906292462 Morse, M. J. (2006). The politics of evidence. Qualitative Health Research, 16, 395–404. https://doi.org /10.1177%2F1049732305285482 Schoonenboom, J. (2019). A performative paradigm for mixed methods research. Journal of Mixed Methods Research, 13(3), 284–300. https://doi. org/10.1177%2F1558689817722889 Shannon-Baker, P. (2016). Making paradigms meaningful in mixed methods research. Journal of Mixed Methods Research, 10(4), 319–334. https:// doi.org/10.1177%2F1558689815575861 Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Sage. Tashakkori, A., & Teddlie, C. (2003). Handbook of mixed methods in social and behavioral research. Sage.

Future Tensions and Design Conversations in the Mixed Methods Field: Section 1 Conclusions J o s é F. M o l i n a - A z o r i n a n d S e r g i F à b r e g u e s

The purpose of Section 1 is to reflect on the history and evolution of mixed methods designs and consider the diversity of past and current design conceptualizations. Undoubtedly, the topic of designs is central within the field of mixed methods research. Over the past three decades, methodologists have indicated the main elements, features, and dimensions of mixed methods designs, providing different perspectives as indicated in the chapters of this Section of the Handbook and in the mixed methods literature. In this Conclusion to Section 1, we would like to highlight and summarize some of the main ideas that the authors of the chapters have indicated as relevant for the future of mixed methods designs. We finish with some concluding thoughts for this section.

SOME IMPORTANT POINTS FOR THE FUTURE OF MIXED METHODS DESIGNS Next, we highlight a few important ideas that the authors of Section 1 chapters consider relevant for the future of mixed methods designs. In Chapter 2, John Creswell and Vicki Plano Clark imagine and highlight several relevant aspects for the future:

flexibility of adapted designs in multiphase and longitudinal projects; families of new types of designs to accommodate the variants that researchers use in practice; validity for types of designs and issues of ethics within the designs’ procedures; development of new joint displays taking advantage of modern technologies; adaptations of diagrams used in other methodologies; and applications of mixed methods to pressing societal problems. These authors advocate the relevance of a typology-based approach to mixed methods design. But to be useful, this approach has to continue to evolve along with the expansion and developments in the field and researchers’ practice. Then, the ideas about mixed methods designs have evolved and will continue to evolve. These authors also recognize several alternative conceptualizations proposed by other authors, and they highlight the importance of flexible and complex designs that account for research practice. In Chapter 3, Joseph Maxwell discusses the history of combining qualitative and quantitative methods, which is much longer than is usually recognized in the mixed methods literature. This author also emphasizes the need for a broader definition of mixed methods design, considering alternatives to design typologies and a wider range of approaches to combining methods in the social

92

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

sciences. Maxwell advocates the use of an interactive approach to conceptualizing mixed methods designs. Moreover, this author considers that a more inclusive concept of design will be positive for the future advancement of the mixed methods field. In this regard, the design of a mixed methods study is not just the prior plan for the research. The design also includes research decisions “on the ground” as the study evolves. This idea is related to the concept of emergent mixed methods designs (Creswell & Plano Clark, 2018) where the use of mixed methods arises due to issues that develop during the process of conducting the research. Donna Mertens, in Chapter 4, points out a key message for the future of mixed methods studies and designs: a call to action with the purpose of solving societal problems, and addressing economic, environmental, and social justice. Using a transformative approach that can provide guidance in the design of mixed methods studies, this author highlights several relevant aspects that must be emphasized in the future of mixed methods designs. Mertens highlights the need for engagement with a broad coalition of stakeholders in culturally respectful ways, including those who are vulnerable and marginalized. To increase positive impact on these stakeholders, a key aspect will be to develop relationships through effective communications throughout the study, and consider these participants as co-researchers. The design of our future mixed methods studies must take into account these and other important aspects indicated in this chapter. In Chapter 5, Katrin Niglas considers that flexible thinking is a key aspect to designing mixed methods studies. This author considers that an important aspect for the future is to encourage mixed methods scholars to adapt and adjust typical designs depending on the practical needs of their research contexts. Multilevel and digital designs will play a key role in future studies and projects, together with innovative and iterative qualitatively driven mixed methods designs. She also emphasizes that wicked problems and grand societal challenges call for new research designs. Niglas also highlights that new tools for integrating data and analyses in mixed methods designs, as well as for visualizing integrated results, will be a key aspect for the future. Another important point for the future is the use of mixed methods designs that are sensitive to diverse cultures. Finally, in Chapter 6, Dawn Freshwater and Jane Cahill highlight the need for future practices in mixed methods designs of a conceptual map of discourse development that can be used to promote awareness in practitioners, researchers, and policy-makers of how discourses surrounding research practices are generated. This conceptual

map of discourse development may help promote an active stance in how a given research paradigm might develop. These authors recommend that the conceptual map can be used and refined considering differing contexts, emphasizing how discourses relating to research evidence and research practices are produced and inform the research design. These authors also indicate that a key point for the future is that critical reflection on discourse development will better enable the research design, the questions being posed, and the findings generated.

CONCLUDING THOUGHTS In our conclusions, we would like to emphasize the relevance of innovations and diversity of perspectives about mixed methods designs, and the key point of tensions and open conversations in the mixed methods community about this central topic. We consider that a dialectical position may strengthen the mixed methods field in this discussion and conversation about mixed methods designs. Greene (2007) pointed out that a dialectical stance in mixed methods research enables an engagement with difference, promoting values of tolerance and acceptance. These values can and must be applied not only to the general acceptance and equity between the quantitative and qualitative components of a mixed methods study, but also to the specific conversations about the different perspectives of mixed methods designs. In our opinion, a key point for the future of mixed methods designs is how different perspectives and views of mixed methods designs can learn and advance from each other. In Chapter 2, Creswell and Plano Clark indicate: “to keep the field open to alternative approaches is useful, and the future needs to embrace multiple ways of engaging in mixed methods research”. The existence of different perspectives and their tensions should stimulate debate and, as noted by Fàbregues et  al. (2021), in the instance of the existence of criticisms in the field of mixed methods, this engagement can lead to valuable insights that contribute to the development of the field. We consider that more emphasis must be put into looking for points of interface and synergies between different and alternative approaches to mixed methods designs. In fact, we would like that these different approaches can be considered as complementary (“both/and” view) rather than alternative and separate perspectives (“either/or” view). We would prefer a “both/and” approach— for example, discussing how a typological

SECTION 1 CONCLUSIONS

approach and an interactive perspective can work together for a better design of a study (both planned and emergent aspects), or how one approach can consider aspects from other approaches. We, as mixed methods methodologists and/or mixed methods empirical researchers, must know and consider the different approaches and perspectives in mixed methods designs. It is important to develop new ideas that can be useful for conducting future mixed methods studies and projects. But it is also relevant to read and know how empirical scholars design, conduct, and implement their mixed methods studies and practices with creativity, innovation, and flexibility, considering the specific contexts of their studies. Together with articles published in regular issues, several journals have also published special issues about mixed methods designs and related topics (see, for example, several special issues and virtual special issues published in the Journal of Mixed Methods Research). As indicated throughout in the Handbook, there are many important aspects to be considered concerning mixed methods designs. In our opinion, as a future important aspect, we would like to highlight that mixed methods designs must take into account in an explicit way how the perspectives of the relevant stakeholders in a research study can be considered in the design and in the research process (Poth et al., 2022). We need mixed methods designs not only for understanding our substantive topics, but also for promoting actions to solve important problems in our society. We need mixed methods designs not only about people, but also with and for people and their problems. We need that mixed methods designs consider how to define the appropriate problems with relevant stakeholders and how to identify actions with these stakeholders that help solve their specific problems in their contexts. We need to use typological approaches, interactive approaches, and other perspectives that help us to promote a better society. As noted above, flexibility, creativity, and innovation in designs will play a key role, and we can improve this flexibility, creativity and innovation through the knowledge of different design approaches and the reading of published papers with innovative and complex designs. We would like to finish by emphasizing that there are several controversies, challenges, criticisms, and tensions in the mixed methods field. Based on previous literature and the opinion of several researchers, Fàbregues et al. (2021) summarized and identified several important criticisms. Many of these criticisms are directly related to the topic of mixed methods designs: limitations of typologies, procedures described in the literature are not aligned with mixed methods practice,

93

the terminology used in mixed methods reflects a lack of agreement among its proponents, and mixed methods research aligns mainly with positivism. The advancement of the mixed methods field in general, and the topic of mixed methods designs in particular, requires intellectual discussions and conversations about these criticisms, tensions and controversies, taking into account the diversity of approaches and perspectives. We need to consider that the mixed methods community reflects a diversity of researchers in terms of disciplines, locations, methodological expertise, and paradigmatic viewpoints. We are sure that the discussions and conversations between members with different backgrounds will help advance our mixed methods field, and the specific and central topic of mixed methods designs. We hope that this section of the Handbook promotes these conversations. We encourage the reading of the chapters in this section and in other sections of the Handbook to know and learn past, current, and future key aspects of mixed methods designs.

ACKNOWLEDGEMENTS We would like to thank Cheryl Poth for the opportunity to collaborate in this section of the Handbook. We are very grateful to Cheryl for her initiative with this publication, and for all her help and support. Her leadership has been a key aspect for the development of this section and the entire project of this Handbook. We would also like to thank the authors of the five chapters (John Creswell, Vicki Plano Clark, Joseph Maxwell, Donna Mertens, Katrin Niglas, Dawn Freshwater and Jane Cahill) for their excellent contributions to this section. Reading their latest ideas about mixed methods designs has been a great learning experience for us. Finally, we are also very grateful to the reviewers of these chapters. They provided interesting comments to improve the chapters.

REFERENCES Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Fàbregues, S., Escalante-Barrios, E. L., Molina-Azorin, J. F., Hong, Q. N., & Verd, J. M. (2021). Taking a critical stance towards mixed methods research: A

94

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

cross-disciplinary qualitative secondary analysis of researchers’ views. PLoS ONE, 16(7). e0252014. https://doi.org/https://doi.org/10.1371/journal. pone.0252014 Greene, J. C. (2007). Mixed methods in social inquiry. Jossey-Bass.

Poth, C., Molina-Azorin, J. F., & Fetters, M. D. (2022). Virtual special issue on “Design of mixed methods research: Past advancements, present conversations, and future possibilities”. Journal of Mixed Methods Research, 16(3), 274–280. https://doi. org/10.1177/15586898221110375

SECTION 2

The Craft of Mixed Methods Research Design

This page intentionally left blank

The Craft of Mixed Methods Research Design: Section 2 Introduction Sophia L. Johnson and Judith Schoonenboom

INTRODUCTION Mixed methods research design has been described as both a process and a product, with the process of design resulting in the design product (Maxwell, 2013; Schoonenboom & Johnson, 2017). A mixed methods design process is not one linear, simple “generation” of a design product. Instead, it is an iterative process, because elements of the design (e.g., points of integration, sampling strategies) might need to be revisited and adapted throughout the study to ensure that they continue to logically work together to address the research question (Schoonenboom & Johnson, 2017). The iterative cycles that are frequently required to design mixed methods research are due, at least in part, to the properties of the methodological tradition: (a) collection and analysis of information from across the data continuum; (b) embrace of diverse philosophical stances and worldviews with the goal of addressing complex, context-specific enquiry; and (c) central but daunting undertakings of integration and developing a meta-inference (Bazeley, 2018; Greene, 2007; Guest, 2012; Johnson, et al., 2007; Schoonenboom, 2022; Teddlie & Tashakkori, 2009). The inherent complexity of mixed methods research in the setting of research environments that are constantly evolving because of new

technology and that are encumbered by globally relevant strife (e.g., pandemics, rapidly deepening social inequality, crisis for climate, and conflict refugees and asylum seekers) has led to calls for mixed methods researchers to become bricoleurs who can approach research design with a “craft attitude” (Bueddefeld, et al., 2021; Crawley, 2021; Denzin, 2012; Oxfam International, 2022; Poth, 2018; Sanscartier, 2020, p. 48). In the six chapters included in this section of the Handbook, the authors respond to the call to apply craftsmanship to the design process and, accordingly, describe innovative practices involved in designing research elements and situating them within larger design products and the iterative mixed methods research design process. Thus, to lay the foundation for a deeper understanding of the approaches set forth within the chapters, we next review the attributes of a craft attitude.

MOTIVATING A CRAFT ATTITUDE Sanscartier (2020), describes a craft attitude as a “collection of specific practices” (p. 48) to address the complexity of designing and conducting mixed methods research, which includes the need

98

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

to make “ad hoc adjustments to research design” (p. 49). He defines the craft attitude, as it relates to mixed methods research, based on a conceptualization by Daft in 1983, as: a disposition (not a paradigm, method, or design type) towards the mixed methods research process that (a) is comfortable with uncertainty, (b) favors non-linearity and recursiveness in research design, and (c) treats research as an exercise in storytelling, about both the research object and our engagement with that object. (p. 53)

He further states that a craft attitude to mixed methods design is buttressed by three cognitive activities: Episteme: “the production of knowledge that is independent of time and space … [which] in the specific field of mixed methods most closely corresponds to typologies”. (p. 51) Techne: “the application of technical knowledge and skills according to a pragmatic instrumental rationality … thus [it] captures the adaptive decision-making process of craftspeople working towards a certain goal in a specific context”. (pp. 51–52) Phronesis: “the deliberation about values with reference to praxis … phronesis concerns the ethics that underpin our actions. (p. 52)

In summary, a craft attitude supports the idea that design processes should reflect ethical practices and established guidance from the mixed methods

field, but must also be creative, flexible and culminate in research with the capacity to tell a story of, for example, the circumstances affecting research participants and their communities. To bolster this claim, we now present an overview of the chapters in this section (see Table S2.1) to illustrate how the authors applied a craftsman approach to designing whole studies (e.g., an emergent design), or specific research elements (e.g., visual displays and meta-inferencing, sampling and ethics, data integration and analysis, ethics and new data sources, integrating methodologies and visual displays).

CRAFTING RESEARCH DESIGNS: APPLICATIONS FROM SECTION 2 In Chapter 7, “Embracing Emergence in Mixed Methods Designs: Theoretical Foundations and Empirical Applications”, De Allegri and Lohmann use the mixed methods literature and the experiences from their research team and the work of their colleagues to define, conceptualize, and document emergent research design practices. The authors acknowledge that there is no universal definition for an emergent approach and sparse reporting of practical matters regarding how to undertake the process. Their view, which represents an expanded definition over prior work, is that emergent designs include adaptations to design elements after a study is underway (e.g., the inclusion of additional data sources or analytic approaches), but that a study may also be envisioned as fluid

Table S2.1  Summary of Section 2 chapters: The Craft of Mixed Methods Research Design Chapter authors (country affiliation)

Chapter title

Manuela De Allegri (Germany) and Julia Lohmann (UK)

Embracing Emergence in Mixed Methods Designs: Theoretical Foundations and Empirical Applications The Methods-Inference Map: Visualizing the Interactions Between Methods and Inferences in Mixed Methods Research Towards Sampling Designs that are Transparent, Rigorous, Ethical and Equitable (TREE): Using a Tree Metaphor as a Sampling Meta-Framework in Mixed Methods Research Data Integration as a Form of Integrated Mixed Analysis in Mixed Methods Research Designs Ethical Issues and Practices for Mixed Methods Research in an Era of Big Data Building the Logic for an Integrated Methodology: Mixed Method Grounded Theory as an Example of Constructing a Methodology to Guide Design and Integration

Judith Schoonenboom (Austria)

Julie A. Corrigan (Canada) and Anthony J. Onwuegbuzie (UK) Susanne Vogl (Germany) Roslyn Cameron and Heinz Herrmann (Australia) Elizabeth G. Creamer, Cassandra McCall, and Cherie D. Edwards (USA)

SECTION 2 INTRODUCTION

from the point of conception. They support the idea that an emergent design is philosophically undergirded by dialectical pluralism, and describe the points along the research process in which emergent elements may be integrated. Importantly, they also discuss potential obstacles to executing an emergent design and how to overcome these barriers. Schoonenboom is co-lead and author of Chapter 8, “The Methods-inference Map: Visualizing the Interactions between Methods and Inferences in Mixed Methods Designs”, where she greatly expands the utility of visualization techniques to support the process of research design. The visual display, as developed by Schoonenboom, goes beyond showing the order of activities in the study. She includes elements of static and interactive visual representations; a comprehensive presentation of research elements; and the links between research strands and inferences that build up to meta-inferences for the whole study. Thus, her approach allows researchers to more exhaustively examine the relationships between design elements and their proposed contribution to research goals. Since her map includes the study population(s), they may also provide the research team with insights about whether subgroup analyses or further studies are warranted, and, thereby, the maps may generate new hypotheses or future inquiries. Corrigan and Onwuegbuzie contribute Chapter 9, “Towards Sampling Designs that are Transparent, Rigorous, Ethical and Equitable (TREE): Using a Tree Metaphor as a Sampling Meta-Framework in Mixed Methods Research”, which offers a framework to guide researchers’ sampling decisions. As discussed by these authors, designing samples in mixed methods research is onerous because it frequently involves complicated sampling schemes that include multiple samples, and because it is underdeveloped in the mixed methods literature. The authors therefore provide rationales to reinforce the need for samples to achieve four key attributes—i.e., transparent, rigorous, ethical and equitable—to produce valid study results that are appropriately representative of and justly work to serve the study community. They use the metaphor of a tree to describe their meta-framework, rooted in critical dialectical pluralism, that supports the interconnected stages of decision-making that are involved in selecting samples that possess the four crucial attributes. They demonstrate the utility of their meta-framework and make it accessible to researchers by applying it to a recent joint project. Authored by Vogl, Chapter 10, “Data Integration as a Form of Integrated Mixed Analysis in Mixed Methods Research Designs”, discusses issues related to the practice of integrating data from across the data continuum into formats and

99

databases that are suitable to conduct joint analysis. As part of this enterprise, she examines the options for integrating data and selects three approaches— i.e., transformation, linkage, consolidation—for further exploration. In her review, she discusses: (a) the benefits and challenges associated with data integration strategies; (b) the analytic directions that may be pursued with the new datasets that are generated from data integration; and (c) the types of research questions that the integrated datasets may answer. Vogl identifies, by applying these strategies to her own empirical work, that a significantly greater understanding of the study phenomenon is gained from analyzing separate and integrated strands of data. Chapter 11, “Ethical Issues and Practices for Mixed Methods Research in an Era of Big Data”, authored by Cameron and Herrmann, examines the ethical issues associated with conducting research utilizing data from new data sources. To provide a foundation for the topic, the authors: (a) review the literature on the ethical considerations associated with designing and conducting mixed methods research; (b) provide a description of the landscape of big data sources; and (c) examine the types of big data used in mixed methods studies to date. They then focus the discussion on the critical ethical dimensions related to the use of big data in mixed methods research—i.e., beneficence, non-malevolence, justice, explicability, and autonomy—by considering their application to two mixed methods studies. They close the chapter by underscoring the need for the mixed methods community to continually examine burgeoning ethical issues as the utilization of big data increases, particularly with respect to the analysis of data released in real time. Creamer, McCall, and Edwards contribute Chapter 12, “Building the Logic for an Integrated Methodology: Mixed Method Grounded Theory as an Example of Constructing a Methodology to Guide Design and Integration”. In this chapter, the authors expound the theoretical underpinnings, rationales, decision-making and practices associated with building an integrated methodology, using grounded theory as an example. Since the goal is to integrate methodologies, they first offer a figure that presents seven levels at which mixing may occur, and this visualization anchors the rest of the discussion. Two mixed methods grounded theory studies are interrogated to illustrate the practices and design decisions associated with integrating at one versus multiple potential mixing levels. Finally, the authors discuss the varieties of grounded theory that may have greater compatibility to integrate with mixed methods research, and provide guidance for supporting the decision to integrate mixed methods with additional methodologies.

100

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

CLOSING THOUGHTS Throughout the six chapters in this section, the authors address underdeveloped and/or novel aspects of mixed methods research. They describe design journeys with open dialogue between research elements. Their discussions reveal iterative design cycles in which they tackle unexpected issues alongside planned activities to responsibly and justly render the meaningful stories evidenced by research results. Thus, in the parlance of Sanscartier (2020), they are approaching design as a craft, and we believe the framing of a crafting process may be useful to advance the field.

REFERENCES Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage. Bueddefeld, J., Murphy, M., Ostrem, J., Halpenny, E. (2021). Methodological bricolage and COVID 19: an illustration from novel, and adaptive environmental behavior change research. Journal of Mixed Methods Research, 15(3), 437–461. https:// doi: 10.1177/15586898211019496 Crawley, H. (2021). The politics of refugee protection in a (post)COVID-19 world. Social Sciences, 10, 81. https://doi.org/10.3390/ socsci10030081 Denzin, N. K. (2012). Triangulation 2.0. Journal of Mixed Methods Research, 6(2), 80–88. https://doi: 10.1177/1558689812437186 Greene, J. C. (2007). Mixed methods in social inquiry. Jossey-Bass. Guest, G. (2012). Describing mixed methods research: an alternative to typologies. Journal of

Mixed Methods Research, 7(2), 141–151. https:// doi: 10.1177/1558689812461179 Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/1558689 806298224 Maxwell, J. A. (2013). Qualitative research design: An interactive approach (3rd ed.). Sage. Oxfam International. (2022). Inequality kills, the unparalleled action needed to combat unprecedented inequality in the wake of COVID 19. https://oxfamilibrary.openrepository.com/bits tream/handle/10546/621341/bp-inequalitykills-1701 22-en.pdf (accessed June, 2022). Poth, C. (2018). The curious case of complexity: implications for mixed methods research practices. International Journal of Multiple Research Approaches, 10(1), 403–411. https://doi: 10.29034/ijmra.v10n1a27 Sanscartier, M. D. (2020). The craft attitude: navigating mess in mixed methods research. Journal of Mixed Methods Research, 14(1), 47–62. https:// doi: 10.1177/1558689818816248 Schoonenboom, J. (2022). Developing the metainference in mixed methods research through successive integration of claims. In J. H. Hitchcock & A. J. Onwuegbuzie (Eds.). Routledge handbook for advancing integration in mixed methods research (pp. 55–70). Routledge. Schoonenboom, J., & Johnson, R. B. (2017). How to construct a mixed methods research design. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(2), 107–131. doi:10.1007/s11577017-0454-1 Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage.

7 Embracing Emergence in Mixed Methods Designs: Theoretical Foundations and Empirical Applications Manuela De Allegri and Julia Lohmann

INTRODUCTION In this chapter, we focus on the opportunity to work with emergent designs, which represents a specific, often neglected, yet key feature of mixed methods research. Although emergent designs are often highlighted as a key feature of mixed methods research (Creswell & Plano Clark, 2017), very few empirical studies are described as having adopted an emergent design. This simple observation, revealing a contradiction between the conceptual and the empirical mixed methods literature, represents the point of departure for our work. We build on an earlier reflection piece (De Allegri et  al., 2020) to advance the conversation and to provide guidance on the adoption of emergent designs in mixed methods research. First, before providing a conceptual and operational definition of emergent mixed methods designs, we offer a critical appraisal of existing literature to identify their origin and delineate their theoretical foundations. Second, based on empirical examples, we take a deep dive into the specific features of emergent mixed methods designs, illustrating how the decision to let a design become emergent can occur at different stages along the research process and can take

different shapes. Last, we examine the challenges that adopting an emergent design can entail and propose strategies to address them.

SITUATING EMERGENT MIXED METHODS DESIGNS IN THE EXISTING LITERATURE Research designs identify the set of procedures used “for collecting, analysing, interpreting, and reporting data in research studies” (Creswell & Plano Clark, 2017, p. 51). Research designs outline the overall strategy guiding the conduct of a study. In line with the epistemology of pragmatism (Johnson & Onwuegbuzie, 2004; Morgan, 2014a), in mixed methods research, all design decisions are oriented towards answering the research question, from the way a study is conceptualized as mixed methods to the relative role and weight assigned to quantitative and qualitative elements, from sampling and data collection all the way to analytical approach and interpretation. Interestingly, the relevant literature does not provide a unanimous definition of what represents an emergent mixed methods design. As outlined

102

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

below, most definitions of emergent designs encompass the concept of change, the possibility that a design of a study is defined or modified in itinere, as the study itself progresses. Creswell and Plano Clark (2017) denote the difference between fixed and emergent designs as a fundamental one to be taken in the conduct of a mixed methods study. Still, their definition of what constitutes an emergent design appears relatively limited, as they define emergent mixed methods designs as designs in which “the use of mixed methods arises due to issues that develop during the process of conducting the research” (Creswell & Plano Clark, 2017, p. 52). Creswell and Plano Clark define as emergent a design where either a quantitative or a qualitative element is added to a single-method study to address an emergent additional concern, usually in response to the limitations of single-method research. By contrast, they define as fixed a design where all quantitative and qualitative elements are planned at the onset of a study. Creswell and Plano Clark acknowledge that the two categories—emergent and fixed—should not be viewed as dichotomous, but as end points of a continuum that serves as the researcher’s orientation when conducting mixed methods research. For a discussion about how their ideas of designs have evolved over time, see also Chapter 2 (this volume). Building on the definition by Creswell and Plano Clark (2017), Schoonenboom and Johnson (2017) posit that the researcher should be open to mixing fixed and emergent elements within a single design, the advice being that of “being prepared for the unexpected” (p. 122). They argue that if, on one side, researchers should ensure robust designs, planning ahead which elements to include in a given study, on the other side, they should also remain open to add elements, as new needs arise during the study execution. Similar to Creswell and Plano Clark, Schoonenboom and Johnson differentiate the concept of complexity from that of the nature of a design, fixed vs. emergent. This distinction emphasizes how both simple and complex mixed methods designs can be either emergent or fixed, suggesting that even a very simple design with only two elements, a quantitative and a qualitative one, can be emergent. Looking further back into early conceptualizations of mixed methods, we recognize that the purpose classification proposed by Greene and colleagues (1989) implicitly opened the way for emergent designs. These researchers distinguish five purposes for choosing a mixed methods design above a single-method one when conducting an evaluation—namely, triangulation, complementarity, development, initiation, and expansion. Although we recognize the potential

for any purpose to lead to an emergent design, we note the specific potential for emergent designs that is built within studies guided by an initiation purpose. With its focus on seeking contradictions and new explanations, the concept of initiation opens the way to recasting research questions as a study unfolds, inevitably calling for changes to be implemented alongside the research process, from data collection to interpretation. Pluye and Hong (2023) build on Greene et  al.’s (1989) initiation purpose to describe an emergent mixed methods design born out of the need to reconcile discordant results, often by initiating a new study or evaluation, addressing questions emerging from this initial discordance. They describe these designs as consisting of a generative process, which can either call for collection of additional data or further analysis of existing data. Pluye and Granikov (2022) also identify in the need to integrate divergent quantitative and qualitative findings an opportunity to engage in an emergent design, specifically what they call the “follow the thread” design. In this design, quantitative and qualitative elements are connected so that evidence emerging from one strain of analysis is used to inform and reshape the analysis carried out in the other strain. The emergent nature lies in the fact that the researcher is called to follow the thread of evidence emerging in one strain to reconsider analytical choices made in the other. Similarly, the interactive approach to mixed methods research proposed by Maxwell and Loomis (2003) also introduces some elements that feed directly into the logic of emergent mixed methods designs. Against the adoption of a typological approach to mixed methods research, Maxwell and Loomis postulate that any study design is constituted of five elements—goals, conceptual framework, research question, methods, and validity — and encourage researchers to constantly check for alignment across these different elements. Their basic argument is that the alignment foreseen by the initial mixed methods design may be lost as the researcher progresses through their work and focuses on single strains of data collection and analysis. Hence, Maxwell and Loomis recognize the need to adapt design decisions to changing circumstances in an interactive process that brings together the different components of a study, explicitly recognizing and adjusting to the influence that one component bears on the others. For a further discussion of an interactive, rather than typological, conception of mixed methods design, see also Chapter 3 (this volume). Beyond these few explicit and implicit accounts, the theoretical and methodological literature is

EMBRACING EMERGENCE IN MIXED METHODS DESIGN

generally silent on the origin and role of emergent mixed methods designs, and provides limited guidance on when and how to resort to an emergent design. Several authors have argued in favour of typological vs. interactive approaches to mixed methods research (Schoonenboom & Johnson, 2017), but this differs from explicitly highlighting the role that emergent designs have to play in mixed methods research. This leaves researchers first approaching mixed methods research at a loss, since they are often unable to gauge if and to what extent it is possible for them to embrace change as a core feature of their design. Moreover, we were not able to locate when and how emergent designs were first conceptualized in the relevant literature.

THE THEORETICAL FOUNDATION OF EMERGENT MIXED METHODS DESIGNS Locating the theoretical foundation of emergent mixed methods designs requires that we first take a look at the Epistemological foundation characterising qualitative and quantitative designs respectively, since mixed methods designs inevitably build on those. Reflecting its root in social constructivism (Andrews, 2012; Creswell, 2013; Walker, 2015) and acknowledging the existence of multiple realities, qualitative research is per se defined as emergent in nature, whereby new ideas, concepts, and findings are integrated to define a study design as the research proceeds (Pailthorpe, 2017). Such an approach to the continuous redefinition of the research process in light of emerging ideas, concepts, and findings is pronounced in some qualitative traditions more than in others—for example, it constitutes the backbone of grounded theory (Glaser & Strauss, 1967), but permeates all qualitative research. Emergent designs are often embedded within qualitative research and integrated at every stage from conceptualization to publication. Qualitative researchers are engaged in a constant process of reflection, allowing themselves to modify what data to collect, how to collect them, how to analyse them, and how to appraise and report them, depending on emerging findings and changing field circumstances. This openness to redefining the different study elements in light of emerging findings and changing circumstances is aligned with the inductive nature of qualitative research (Morgan, 2014b, p. 45). By contrast, reflecting its root in positivism, quantitative research is normally characterized by

103

fixed designs, whereby strategies for data collection and analysis are set in advance and normally not modified during the course of the study. This approach is consistent with the deductive nature of quantitative research, testing theories through the systematic observation and measurement of reality (Morgan, 2014b, p. 45), the underlying assumption being that reality is sufficiently stable to be observable and measurable. When conducting a quantitative study, researchers are called to formulate their hypotheses explicitly to be able to identify in advance the most suitable data collection and data analysis strategies. More recently, however, quantitative research has opened to the inclusion of more flexible, if not emergent, designs. In particular, we note the increasing application of exploratory quantitative approaches, whereby data-collection strategies are normally still fixed in advance, but analytical approaches may be redefined as the analysis proceeds (Ferketich & Verran, 1986). Adaptive designs in clinical trials offer an even more notable example of the increased flexibility recently embraced by quantitative research (Pallmann et al., 2018). We postulate that the integration of emergent designs in mixed methods research can largely be traced back to its close relationship to qualitative epistemological approaches and practices. Pragmatism, as the epistemological foundation at the core of mixed methods research (Morgan, 2014a; Johnson & Onwuegbuzie, 2004), opens the way to the adoption of emergent design. It does so by inviting researchers to be open to any epistemological approach and hence to any design, and any method, which is most suited to answer a given research question. This includes embracing the possibility that some design decisions may not be clearly defined in advance and may only emerge as a study progresses. We recognize, however, that researchers with limited exposure to the epistemological foundations of qualitative research and to its practice may perceive this invitation as a challenge, since it represents a major departure from the deductive tradition of carefully planning studies in advance to test clearly framed hypotheses. Hereafter, we illustrate how embracing a dialectical approach can turn this challenge into an opportunity, while keeping true to pragmatism as the leading epistemological tenet in mixedmethods research. We argue that by embracing dialectic pluralism and with it the option to let designs evolve alongside the conduct of a study, we can deliver higher quality and more responsive mixed methods research, providing more credible answers to the most pressing questions of our times.

104

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Dialectic Pluralism as the Foundation of Emergent Designs In its original meaning, the term “dialectics” was used by Greek philosophers to refer to a back-andforth dialogue or debate between people holding opposing points of view. In the 19th century, Hegel furthered the understanding of the term to refer to opposing sides, not only as different sets of people advancing different points of view on a given topic, but also as opposing definitions of logical concepts (Maybee, 2020). By recognizing the existence of three moments in the development of a concept – namely, the moment of understanding, the dialectical moment, and the speculative moment, Hegel postulated that unity emerges out of the contradiction and resolution of ideas, and the relationship between them. While it is beyond the scope of our chapter to examine the philosophical underpinning of the term, it is important to understand its origin to locate its more practical adoption in mixed methods research, where a dialectical and dialogical approach is used to reflect the extent to which one puts different methods and findings into dialogue with one another (Taylor & Raykov, 2020), letting concepts emerge out of a process of reconciling initial contradictions. Dialectical pluralism is increasingly being recognized as a guiding principle in mixed methods research, which instead of silencing differences in epistemological and methodological traditions, thrives on them to generate better research outcomes (Johnson, 2015; Johnson et al., 2014). Dialectical pluralism explicitly recognizes tension points between the different traditions that come together in mixed methods research, and encourages the adoption of a dynamic approach to resolving them by integrating different perspectives in innovative ways, to generate knowledge that could have not been produced in the absence of these initial apparent contradictions (Johnson, 2015). We do not argue that emergent designs represent the only way forward when we embrace dialectical pluralism. We recognize that it is possible to adopt a dialectical approach to mixed methods research, keeping the dialogue open between quantitative and qualitative strains of analysis, even in the presence of a fixed design. However, we do identify in the dialectical approach to mixed methods a further development of the interactive approach, whereby the researcher engages not only in a reflection on the single study components, but in an active dialogue across methods and findings, in a way that largely resembles the moments described by Hegel to come to identify and define concepts. Once a researcher opens the

door to dialectical pluralism, they inevitably also open the door to emergent designs, since accepting to see and resolve tension points in their work will inevitably lead them to identify the need to adjust design and methodological decisions alongside the conduct of a study. It is essential to note here that adopting a dialectical approach does not entail abandoning advanced planning, but exclusively remaining open to the option that the best way to answer a given research question may require a different approach from the one originally planned. In some instances, when exploring new grounds or working under uncertainty, the initial study plan may entail an emergent design, recognizing that methodological decisions can only be made in itinere.

OPERATIONAL DEFINITION OF EMERGENT MIXED METHODS DESIGNS In line with these conceptual postulations and expanding our prior operational definition (De Allegri et al., 2020), we define as emergent any design that is born out of an iterative dialogue between quantitative and qualitative strains of data collection and analysis, allowing for revisions in the conceptualization, sampling, data collection, and data analysis and interpretation, as the study unfolds. We define emergent designs as designs that are responsive to changing research needs, as shaped by field circumstances as well as by researchers’ evolving thoughts. Our operational definition expands, but does not per se contradict, the understanding of emergent designs proposed by Creswell and Plano Clark (2017), which refers to the addition of a qualitative or quantitative element to a planned singlemethod study, nor those proposed by Pluye and colleagues (in press), which refer to specific instances calling for a design to be modified. We explicitly recognize that the decision to let a design be shaped by changing research circumstances can come up at different times and logical points along the conduct of a study and can take different shapes. We further recognize that emergent designs exist in their own right, not just in juxtaposition to fixed designs. This is to say that while at times modifications are applied to turn a design from fixed into emergent, at other times, designs are conceived as emergent from their very onset. Finally, we recognize that revisions in the context of emergent design may be of different magnitudes, depending on a study’s specific needs.

EMBRACING EMERGENCE IN MIXED METHODS DESIGN

In the following sections, we examine, drawing from empirical examples, the practicalities of engaging with emergent designs, including the circumstances that may call for their adoption and the challenges associated with them.

EMBRACING EMERGENCE AT DIFFERENT STAGES OF THE RESEARCH PROCESS Hereafter, we first lay out what defines adopting an emergent approach at each of the above-mentioned research stages – namely, conceptualization, sampling, data collection, and data analysis and interpretation, and then identify what circumstances may call for such an approach to be implemented (Table 7.1). We have decided to bind the description of the

105

revisions that can be implemented to that of the circumstances that may call for them, because it is our experience that the two are often closely related. It is our intention to enrich our writing with empirical illustrations, capturing researchers’ real-life experiences. Given the paucity of written accounts of emergent designs, we have often resorted to documenting our own unpublished experiences or those of close colleagues.

Emergence at the Conceptualization Stage We use the term “conceptualization” to identify the earliest stage in a research project, the moment when a researcher is concerned with framing the

Table 7.1  Synthesis of the practice of embracing emergence in mixed methods research Conceptualization

Sampling and data collection

Data analysis and interpretation

What brings us to embrace emergence at the level of . . .

The need to embrace emergence at the conceptualization stage, either by leaving mixed methods designs open to be defined along the study process or by implementing changes to an originally planned design, often arises when addressing complex research questions and/ or wishing to address new questions that arise along the way.

The need to embrace emergence at the sampling and datacollection stage often arises either in response to changes at the conceptualization level or in response to additional data sources becoming available.

The need to embrace emergence at the analytical changes may result from changes being implemented in prior stages, but also in their own right in response to emerging findings in other study components, or in response to changing policy needs, leading to modifications in the research questions being addressed.

We embrace emergence by . . .

Including additional quantitative or qualitative components. Changing the relative weight of each component within the study. Changing the point of integration.

Including an additional data source or an additional sample. Adding or changes in datacollection tools Implementing major change in data-collection procedures.

Shifting analytical focus or technique. Changing points of integration between quantitative and qualitative findings.

The added value resulting from embracing emergence is . . .

The ability to flexibly react to changing societal and policy concerns, thereby increasing the relevance and importance of on-going research.

The ability to generate more comprehensive and credible evidence by expanding data collection. The ability to flexibly react to emerging needs to triangulation or validation.

The ability to react to unexpected emerging research questions, often based on counterintuitive or contradicting emerging findings. The ability to enhance integration across quantitative and qualitative findings.

106

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

research question and defining their overall research strategy. This is the moment when the researcher first conceptualizes how the different quantitative and qualitative elements will come together in a single design to answer a given research question, and makes decisions on sequence, relative weight and points of integration. Emergence at the conceptualization stage involves opening the way to substantial design changes being made in response to emerging opportunities or challenges being anticipated or being encountered along the conduct of a study. More specifically, at times, changes can be partially anticipated, meaning that a researcher initiates a study, already knowing that additional components will need to be integrated in light of emerging research needs. At other times, these changes cannot be anticipated and are implemented just in response to changing research circumstances. It is our experience that narrowly defined research questions can more easily be addressed by fairly fixed designs, while more complex research questions often call for emergent designs to be anticipated already at the conceptualization stage. Both Creswell and Plano Clark’s (2017) description of emergent designs and Pluye and Hong’s (in press) initiation design refer to cases where the emergent element is incorporated at the conceptualization stage, through the inclusion of an additional quantitative or qualitative component, complementing findings from a study initially planned otherwise. The evaluation literature has long recognized the need for emergent designs to be considered as the most viable research option already at the very onset of a study (Christie et al., 2005). This explicit recognition of the role of emergent designs in evaluation stems out of the need to work alongside the implementation of an intervention or program, without necessarily being able to anticipate all research questions and knowledge needs at the beginning of the research process. Christie et al. (2005) provide an accurate account of how the evaluation of the Teaching and Learning Communities Program was explicitly conceptualized as an emergent design, with shifts in inquiry focus and related data collection and analysis methodology being anticipated, but not defined, from the onset of the research process. Similarly, the evaluation focus of our work in global health has often led us and our colleagues to conceptualize our studies as emergent already at the onset of the inquiry process. The most prominent illustration comes from work we have recently initiated in Pakistan, where, in collaboration with colleagues at the University of Erlangen and at Khyber Medical University, we

are asked to monitor the medium- and long-term impacts of introducing health insurance coverage for outpatient health care services on both beneficiaries’ and health care providers’ outcomes. Since not all elements of the intervention had been defined at the onset of the research project, we have opted to adopt an emergent mixed methods design. This means that, while on one hand we have defined broad research questions and related quantitative and qualitative methods of data collection and analysis, on the other hand, we have left room for adaptation in our design. We have made the emergent nature of our design explicit with both concerned policy makers and funding agencies, motivating it in relation to the need to adjust the content of our research to the actual program evaluation needs. Obtaining funding for our work has not been challenging in this instance, because we engaged in an extensive dialogue with concerned stakeholders beforehand to illustrate the benefits of being explicit about the adoption of an emergent design. This resulted in a collective buy-in by all concerned stakeholders. Busetto and colleagues (2017) also provide a detailed account of how they were unexpectedly called to redefine their study design at the conceptualization stage, but only once the study had already been initiated. They had initially planned to examine workforce changes in integrated care interventions through the application of a literature review to be followed by a quantitative check of emerging findings via a Delphi panel. Once these researchers realized that the results of the literature review were not appropriate for quantitative confirmation, they adjusted their study design to include, in place of the Delphi panel, a qualitative exploration and secondary analysis of two best practice case reports. It is our experience that, especially for researchers engaged in evaluation across policy sectors, design revisions at the conceptualization stage often emerge out of a need to adjust to changing societal and policy concerns. This has become very apparent over the course of the last two years in relation to the spread of the SARS-CoV-2 pandemic. Drawing from our own experience, for instance, we have often had to modify the conceptualization of our ongoing studies on health financing and on health service provision to accommodate the need to address the overwhelming impact that the pandemic is bearing on health systems. These changes have ranged from including additional quantitative or qualitative components to changing the relative weight each component would have within a study to changing the point of integration between the two.

EMBRACING EMERGENCE IN MIXED METHODS DESIGN

Emergence at the Sampling and Data Collection Stage We identify as defining elements of an emergent mixed methods design any substantial revisions to sampling and data collection strategies that are implemented in response to changing research needs. We recognize that while all changes at the conceptualization stage normally also entail changes in sampling and data collection, not all changes in sampling and data collection entail changes at the conceptualization stage. Moreover, we acknowledge that small changes to sampling and data collection, such as increasing or decreasing the exact numbers of respondents in a survey or redefining some questions in an interview guide, are inevitable in most studies, whether single- or mixed methods, and are alone not sufficient to identify a design as emergent. While we are aware that there is no single threshold that allows us to designate a design as emergent, we generally recognize as emergent a design where substantial changes to sampling and data collection have been made—for instance, through the inclusion of an additional data source or an additional sample, ultimately leading to an additional analysis as well. Most revisions that are implemented at this level occur when a researcher encounters new opportunities for data acquisition or collection. Our work in Malawi illustrates this case (De Allegri et al., 2020). We were tasked with assessing changes in utilization of obstetric services following the introduction of the Results Based Financing for Maternal and Newborn Health (RBF4MNH) Initiative in four districts, and were puzzled by the discordance between quantitative and qualitative results. On one hand, quantitative findings indicated that utilization of obstetric services had not increased among pregnant women following the introduction of the RBF4MNH Initiative. On the other hand, health care workers at RBF4MNH actively reported an increased workload, due to more women visiting their facilities for obstetric services. Without changing the overall structure of the study design—i.e., without applying substantial modifications to conceptualization of the quantitative and qualitative elements included in the study—we turned to an additional quantitative data source to unravel sources of discordance between our quantitative and qualitative findings. More specifically, our quantitative component relied on the analysis of primary data on health service utilization collected via means of a household survey on a sample of nearly 2000 households. This sample, however, turned out not to be sufficiently large to allow us to detect the relatively small changes in utilization patterns that the intervention was producing. So, once

107

we learned that the quality of the routine health management information system data had radically improved over the last few years, we turned to these data to examine utilization patterns and triangulate our analysis based on primary household survey data. The inclusion of the additional dataset and analysis did not per se call for modifications of our overall design, although other elements later did, but allowed us to strengthen the quantitative component by triangulating primary findings and generating more credible results. Had we not become aware that such data were of good quality and available to researchers upon request, we might never have solved the initial discordance between quantitative and qualitative findings. Changes in sampling and data collection may also be motivated by a renewed understanding of the issue under investigation. This renewed understanding can be stimulated either by preliminary findings emerging from one of the two strains of analysis, or by acquiring access to additional information on the sociocultural context within which the research takes place. McMahon and colleagues (2015) provide an illustration of how emergent findings from the quantitative strains called for revisions to be applied to sampling and data collection in the qualitative strain. Their study aimed to assess and understand early discharge after delivery in rural Tanzania. Although their study is presented as a mixed methods design fully planned in advance, discussion with the authors revealed that while the initial plan was to conduct interviews exclusively with women who had delivered in the prior 14 months, results emerging from the preliminary quantitative analysis revealed the need to discuss also the time to discharge with women’s husbands and with community leaders. In addition, what is not reported in the publication by McMahon et  al. (2015), but also speaks of the emergent nature of mixed methods designs, is the fact that the authors ultimately also reached out to health care providers to capture their perspective on early discharge, largely out of the need to validate evidence generated at the population level vis-àvis the national health authorities. Effectively, their approach to sampling and data collection changed substantially, as the authors acquired new information on the study context, but without ever challenging the overall conceptualization of the study—i.e., the relative weight and the point of integration of qualitative and quantitative findings.

Emergence at the Data Analysis and Interpretation Stage We define a design as emergent when substantial changes at the data analysis stage are

108

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

implemented, often as a consequence of the mixed methods nature of the study—that is to say, analytical changes in one strain of analysis emerge in response to what findings are emerging in the other strain. We acknowledge that changes at the analytical level are often also induced by changing policy needs, most frequently in relation to evaluation research, as well as by requests made by funding and implementing partners. Similar to what was described earlier in relation to sampling and data collection, we describe a design as emergent only when these changes are substantial, calling for fundamental revisions of the analytical plan originally developed. What Pluye and Granikov (2022) describe as the “follow-the-thread” approach represents an excellent illustration of how the emergent element often arises at the analytical stage. In line with our understanding that emergent designs are born out of the dialectical element of mixed methods research, Pluye and Granikov describe how Boot and colleagues (2016) revised the analytical strategy for their cohort of individuals aged 58 to 65 and affected by either depressive disorder, cardiovascular disease, or osteoarthritis, in light of qualitative findings suggesting that factors they had not considered in their initial analysis might be relevant to examine work participation. It is not only the addition of a few variables that renders the work by Boot and colleagues emergent, but rather the fact that these variables build on new emergent hypotheses that in turn are informed by emergent qualitative findings. Similarly, our mixed methods impact evaluation of the RBF4MNH Initiative in Malawi (De Allegri et al., 2020) can be referred to as a clear “followthe-thread” example. It was by following up on the initial discordance between quantitative and qualitative findings describing women’s patterns of utilization for obstetric services that we identified the potential for the intervention to have changed referral patterns in control more than in intervention facilities. In response, we decided to follow up, with a set of dedicated hypotheses and additional ad hoc analyses, conducted to address this emergent research question. Our work on RBF4MNH also provides an example of how revisions to the analytical approach can be stimulated by requests coming by the funding and/or implementing partners. We worked on the project for over six years, but initially we had not planned to use data at our disposal to look into the program impact on maternal mortality, assuming that the observation period was too short to allow for such analysis. However, we felt compelled to attempt such an analysis, following an explicit request by implementing health care providers, who insisted that they were witnessing fewer maternal deaths, but did not know how

to demonstrate the validity of their claim to policymakers (De Allegri et al., 2019). We explicitly acknowledge that embracing revisions at the analytical stage inevitably leads to revisions in the interpretation and reporting of the overall study, since it likely leads to modifications in the underlying narrative of the study. Changing analytical approaches brings additional or different evidence to light, inevitably leading to changes also in the way that quantitative and qualitative evidence are integrated into a single narrative (Fetters et al., 2013). In the case of RBF4MNH, for instance, the inclusion of the mortality analysis substantially changed the overall narrative of the evaluation. The changes we detected in facility-based maternal mortality at birth were of a larger magnitude than what we would have expected in light of the changes in the quality of service delivery that we had documented in prior quantitative strains of analysis. The need to explain the new unexpected finding on maternal mortality challenged the dominant interpretation of the findings, forcing us to reconsider emerging hypotheses and points of integration, often returning to the data, both quantitative and qualitative, for further analyses.

Emergence as an Added Value We close this section by bringing the reader’s attention to the fact that in practice many emergent mixed methods designs are characterized by emergent elements being integrated at different stages along the research process. Our description purposely addresses each stage separately, since it aims at highlighting the circumstances that may motivate us to adopt an emergent design, drawing from empirical examples. In practice, however, emergent elements often carry from one phase into the next. For instance, adopting an emergent design at the conceptualization stage may call for the adoption of an emergent approach, also at the sampling and data-collection stage, or at the analytical and interpretation stage. Effectively embracing the emergent nature of mixed methods designs entails acceptance of the fact that research may not proceed along a linear path, but rather take a few loops before being able to fill existing knowledge gaps. Irrespective of the stage where emergence was integrated in study design, all the cases documented in this section point at the added value resulting from embracing emergence as a fundamental feature of our work. In all instances, this choice, implemented either by adopting an emergent design in the first place or by applying major modifications to planned sampling, data collection

EMBRACING EMERGENCE IN MIXED METHODS DESIGN

and data analysis resulted in richer findings. In turn, richer findings allowed researchers to generate more comprehensive and credible evidence to meet the research objectives. This was possible because deploying an emergent design facilitated the integration of data sources and analytical strategies initially not considered by the study teams, but in fact essential to answer either the initial or additional research questions identified along the research process. In closing this section, we wish to turn full circle and point explicitly at how in all instances documented here, embracing emergence followed from embracing dialectical pluralism – that is to say, engaging in a constant and open dialogue across strains of data collection and analysis all along the research process.

OVERCOMING CHALLENGES RELATED TO EMERGENT MIXED METHODS DESIGNS If on one hand there is no doubt that turning to an emergent mixed methods design adds value to our research, on the other hand, there is also no doubt that this choice carries with it an additional set of challenges. Hereafter, we introduce some of the key challenges as well as relevant strategies to address them.

Challenges Related to Validity and Credibility Similar to what has been written in relation to adaptive clinical trials (Pallmann et  al., 2013), researchers embracing emergent mixed methods design need to find ways to embrace flexibility without jeopardizing the validity and credibility of their work. In line with our argument on dialectical pluralism, this means that researchers need to stay alert to reflect and act on any potential challenge to integrity and validity induced by embracing emergence along the research process. While we acknowledge that it is easier to account for the integrity and validity of a study when all elements are planned in advance, we also recognize there can be no trade-off between flexibility and rigor. Both planning for emergence at the onset of a study, as we have done in our work on health insurance in Pakistan, and embracing emergence in itinere, as we have done in our work on the RBF4MNH Initiative in Malawi, did not compromise the study quality. All design, sampling, data collection, analysis, and integration decisions should always be made in respect of the

109

methodological principles that characterize the quantitative and the qualitative sciences. We refer our readers to existing literature to identify the specific strategies available to ensure credibility and validity in mixed methods studies (Onwuegbuzie & Johnson, 2006), as those also apply to emergent designs.

Challenges Related to Resource Availability In line with what has been described by Jamila Harris (2021) reflecting on her own experience of conducting mixed methods research in lowresource settings, researchers need to be aware that design decisions are inevitably shaped by the time, the material, and the human resources available to the team. Discussions with colleagues have revealed that emergent designs are very often resource-intensive designs. Additional resource needs emerge most notably, but not exclusively, when a completely new element is added on to an existing study. Even when revisions are relatively small, however, as one may consider the sample revisions applied by McMahon and colleagues in their work on delivery in Tanzania (McMahon et  al., 2015), one needs to account for the additional time and resource demands. Costs are often the first barrier researchers face in a world where research is so dependent on third-party funding, with strict budgetary allocations defined a priori. We have recently worked on a complex mixedand multi-method Covid-19 surveillance trial (Deckert et  al., 2021), which, over the course of one year of funding, has required us to submit three budget reallocation requests to the funding agency to redirect budget so that we could adjust design to the emergent nature of the pandemic. These administrative procedures absorbed extensive time and energy, effectively reducing the resources available for the research itself. Even when no budget constraints are present and researchers are free to reallocate funding to meet emergent study demands, teams often face time and human capacity constraints. Emergent designs inevitably eat up more time and are difficult to operationalize if a team is bound to tight timelines. Most important of all, however, embracing dialectic pluralism entails opening up to the possibility that answering an emergent research question may require the integration of a methodological approach beyond the team’s capacity. In practice, this may require asking additional researchers with a narrowly defined expertise to join an existing team. While being experts in their specific fields, these researchers may lack

110

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

exposure to mixed methods research and its fundamental epistemological and empirical underpinnings rooted in pragmatism (Feilzer, 2009). It is our experience that the challenges that may arise in these instances, when we are called to collaborate with researchers from very different scientific traditions and who may have never been exposed to mixed methods, are more easily managed when at least one or two senior team members have extensive mixed methods experience and can mediate exchanges within the team, nourishing the dialectic pluralism we referred to earlier. Unfortunately, no course offers training on the practicalities of contributing to mixed methods research in highly heterogenous teams. Such coordination challenges arise both in fixed and in emergent studies, but are most pronounced in the latter, where opportunities to identify and address in advance opportunities and threats are limited. It is our experience that embracing the qualitative call for reflexivity (Dodgson, 2019), examining one’s standing visà-vis the research being conducted, can support researchers as they manage the challenges that arise in coordinating highly heterogeneous teams working with emergent designs. The ability to generate consensus around an emergent design by engaging all concerned stakeholders through a collaborative process has been noted in our own work (De Allegri et  al., 2020) as well as in that of others (Christie, et al., 2005; Tom, 1996) as a key feature to successfully embrace emergence in mixed methods research.

Challenges Related to Feasibility Considerations We recognize that embracing emergent designs can be perceived as audacious, since it presents concrete risks for researchers. Across fields of enquiry, at the proposal writing stage, researchers are normally expected to outline their planned methodology in detail, including samples and data-collection strategies and to indicate the specific point of integration between quantitative and qualitative elements. In proposal writing, there is normally little room to embrace emergent designs explicitly, even when we know from the very beginning that an emergent design would be the most suitable to pursue a given research question. Reflecting back on our own experience, we note that a key success factor for our proposals—those explicitly integrating emergence as a design element – has been the ability to motivate the need to maintain flexibility across design elements in light of the study content, context, and/or resources available. To signal to the reviewers that beyond

flexibility, we have in fact thought through alternative methodological pathways, we describe the details of multiple alternatives, without committing to any. Moreover, while most disciplines nowadays require ethical clearance prior to initiating any study, it is virtually impossible for ethics committees to just sign off on emergent designs, since they cannot anticipate all potential risks to study participants/respondents (Tolich, 2016). This limitation should not act to discourage researchers from engaging in emergent designs, since most ethics committee always consider requests for amendments when departure from the original design are needed and are normally ready to process them rapidly, not to halter research which is already ongoing. Alternatively, when at least some advance planning is possible, researchers can describe in their protocols for submission to ethics committees the different probable data collection and analysis options available under different research scenarios. Last, at least in our field of research, scientific manuscripts are expected to rely on a linear description of the methods implemented. Instead of capitalising on its value, we hide the emergent nature of our designs, as if we feared being judged for not being sufficiently rigorous when describing the loops that accompany our everyday design decisions. We do not mean to say that journal editors are prescriptive in their guidelines for authors and ban inclusion of emergent designs a priori. Prescription rather emerges implicitly when one is confronted with tight word limits or when one feels compelled to comply with the traditional structure proposed by prior publications.

THE FUTURE OF EMERGENT MIXED METHODS DESIGNS Without aiming to be exhaustive on the matter, this chapter has aimed at triggering the reader’s curiosity and at motivating them to embrace emergent designs in their mixed methods research practice. While surely not without challenges, embracing emergent designs when the research question and circumstances make it relevant to do so can be a very fulfilling experience. On the one side, emergent designs allow mixed methods researchers to stay true to their pledge for pragmatism, maintaining an exclusive focus on the need to answer the research question in the most comprehensive and credible manner. On the other side, emergent designs allow mixed methods r­ esearchers to fully entertain their academic selves, feeding

EMBRACING EMERGENCE IN MIXED METHODS DESIGN

their dialectic pluralist personality by opening new venues for research. We trust that, as mixed methods become even more prevalent across fields of inquiry, an increasing number of researchers will inevitably embrace emergence as a key tenet of their designs. In turn, we trust that this increased adoption of emergent designs will stimulate further reflections at conceptual level, ultimately resulting in advances in research practice. The paucity of theoretical and empirical guidance which characterizes emergent mixed methods research at the moment will be overcome as researchers “learn by doing”. This “learning by doing” approach sits at the core of many of the experiences described in this chapter, whereby the reflection on the value to embrace emergence was actually stimulated by the need to do so given very tangible research challenges. Last, we also trust that as the academic community becomes more conversant with the need to remain open to embrace emergence alongside the conduct of a research project, the institutions that support this inquiry process, such as funding agencies, publishers and ethics committee, will also follow on this path. We can imagine a future where researchers won’t be penalized for being open about the possibility to plan each and every aspect of research in advance.

111

Health Policy Planning, 35(1), 102–106. https:// doi.org/10.1093/heapol/czz126

This article provides an illustration of how a mixed methods design emerged in response to changing research needs and available data, outlining both the conceptual and the practical steps that characterized the authors’ work. Pluye, P., & Granikov, V. (2022). Combinations of practical integration strategies used in mixed methods information studies. In P. Ngulube (Ed.), Handbook of research on mixed methods research in information science (pp. 70–86). IGI Global. https://doi.org/10.4018/978-1-7998-8844-4. ch004

This book chapter provides an illustration of the “follow the thread” approach, outlining how one strain of analysis is used to inform and reshape the analysis carried out in the other strain. Pluye, P., & Hong, Q. N. (2023). Convergence and divergence in mixed methods research. In R. Tierney, F. Rizvi and K. Ercikan (Eds.), International encyclopedia of education (4th ed.), Volume 12 ‘Qualitative, Multimethod, and Mixed Methods Research’ (N. V. Ivankova, Ed.), pp. 1–16. Elsevier.

This book chapter provides an illustration of how emergent designs can emerge out of the need to reconcile initial discordance between qualitative and quantitative findings.

ACKNOWLEDGEMENTS We are grateful to all the colleagues who supported us as we wrote this chapter, by engaging in fruitful conversations with us and/or by sharing their experiences conducting emergent mixed methods design. In alphabetical order, we thank: Emmanuel Bonnet, Stephan Brenner, Andreas Deckert, Laura Di Lorenzo, Jamila Harris, Zohaib Kahn, Andreas Landmann, Shannon McMahon, Pierre Pluye, Atonu Rabbani, Valéry Ridde, and Malabika Sarker. We are indebted to Alfonso Valenzuela Hurtado for his support managing references and Felix Amberg for his support with the final editing.

WHAT TO READ NEXT De Allegri, M., Brenner, S., Kambala, C., Mazalale, J., Muula, A. M., Chinkhumba, J., Wilhelm, D., & Lohmann, J. (2020). Exploiting the emergent nature of mixed methods designs: Insights from a mixed methods impact evaluation in Malawi.

REFERENCES Andrews, T. (2012). What is social constructionism? Grounded Theory Review: An International Journal, 11(1). http://groundedtheoryreview.com/2012/ 06/01/what-is-social-constructionism/ Boot, C. R. L., de Kruif, A. T. C. M., Shaw, W. S., van der Beek, A. J., Deeg, D. J., & Abma, T. (2016). Factors important for work participation among older workers with depression, cardiovascular disease, and osteoarthritis: A mixed method study. Journal of Occupational Rehabilitation, 26(2), 160–172. http://doi.org/10.1007/s10926015-9597-y Busetto, L., Luijkx, K., Calciolari, S., González-Ortiz, L. G., & Vrijhoef, H. J. M. (2017). The development, description and appraisal of an emergent multimethod research design to study workforce changes in integrated care interventions. International Journal of Integrated Care, 17(1), 1–1. http://doi.org/10.5334/ijic.2510 Christie, C., Montrosse-Moorhead, B., & Klein, B. (2005). Emergent design evaluation: A case study.

112

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Evaluation and Program Planning, 28(3), 271–277. http://doi.org/10.1016/j.evalprogplan.2005.04.002 Creswell, J. (2013). Qualitative inquiry & research design: Choosing among five approaches. Sage. Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). Sage. https://doi.org/10.1111/j.1753-6405.2007. 00096.x De Allegri, M., Brenner, S., Kambala, C., Mazalale, J., Muula, A. M., Chinkhumba, J., Wilhelm, D., & Lohmann, J. (2020). Exploiting the emergent nature of mixed methods designs: Insights from a mixed methods impact evaluation in Malawi. Health Policy Planning, 35(1), 102–106. https:// doi.org/10.1093/heapol/czz126 De Allegri, M., Chase, R. P., Lohmann, J., Schoeps, A., Muula, A., & Brenner, S. (2019). Effect of results-based financing on facility-based maternal mortality at birth: An interrupted time-series analysis with independent controls in Malawi. BMJ Global Health, 4(3), e001184. http://dx.doi. org/10.1136/bmjgh-2018-001184 Deckert, A., Anders, S., De Allegri, M., Nguyen, H. T., Souares, A., McMahon, S., Meurer, M., Burk, R., Sand, M., Koeppel, L., Hein, L. M., Roß, T., Adler, T., Siems, T., Brugnara, L., Brenner, S., Herbst, K., Kirrmaier, D., Duan, Y., Ovchinnikova, S., … Denkinger, C. (2021). Effectiveness and cost-effectiveness of four different strategies for SARS-CoV-2 surveillance in the general population (CoV-Surv Study): Study protocol for a two-factorial randomized controlled multi-arm trial with cluster sampling. Trials, 22(1), 1–9. https://doi. org/10.1186/s13063-020-04982-z Dodgson, J. E. (2019). Reflexivity in qualitative research. Journal of Human Lactation, 35(2), 220–222. http://doi.org/10.1177/08903344198 30990 Feilzer, M. (2009). Doing mixed methods research pragmatically: Implications for the rediscovery of pragmatism as a research paradigm. Journal of Mixed Methods Research, 4(1), 6–16. http://doi. org/10.1177/1558689809349691 Ferketich, S., & Verran, J. (1986). Technical notes: Exploratory data analysis – Introduction. Western Journal of Nursing Research, 8(4), 464–466. http:// doi.org/10.1177/019394598600800409 Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs – Principles and practices. Health Services Research, 48(6pt2), 2134–2156. http://doi. org/10.1111/1475-6773.12117 Glaser, B. G., & Strauss, A. L. (1967). The discovery of Grounded Theory: Strategies for qualitative research. Aldine. https://doi.org/10.4324/9780203793206 Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational

Evaluation and Policy Analysis, 11(3), 255–274. http://doi.org/10.3102/01623737011003255 Harris, J. (2021). Mixed methods research in developing country contexts: Lessons from field research in six countries across Africa and the Caribbean. Journal of Mixed Methods Research, 16(2), 165–182. Johnson, R. B. (2015). Dialectical pluralism: A metaparadigm whose time has come. Journal of Mixed Methods Research, 11(2), 156–173. http://doi. org/10.1177/1558689815607692 Johnson, R., & Onwuegbuzie, A. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26. http://doi.org/10.3102/00131 89X033007014 Johnson, R. B., Onwuegbuzie, A. J., Tucker, S. A., & Icenogle, M. L. (2014). Conducting mixed methods research: Using dialectical pluralism and social psychological strategies. In P. Leavy (Eds.), The Oxford handbook of qualitative research. (pp. 557–578). Oxford University Press. http://doi. org/10.1093/oxfordhb/9780199811755.013.022 Maybee, J. E. (2020). Hegel’s dialectics. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy. https://plato.stanford.edu/archives/win2020/ entries/hegel-dialectics/ Maxwell, J., & Loomis, D. (2003). Mixed method design: An alternative approach. In A. Tashakkori & C. Teddle (Eds.), Handbook of mixed methods in social and behavioral research (pp. 241–272). Sage. McMahon, S. A., Mohan, D., LeFevre, A. E., Mosha, I., Mpembeni, R., Chase, R. P., Baqui, A. H., & Winch, P. J. (2015). “You should go so that others can come”: The role of facilities in determining an early departure after childbirth in Morogoro Region, Tanzania. BMC Pregnancy and Childbirth, 15(1), 328. http://doi.org/10.1186/ s12884-015-0763-1 Morgan, D. L. (2014a). Pragmatism as a paradigm for social research. Qualitative Inquiry, 20(8), 1045– 1053. http://doi.org/10.1177/1077800413513733 Morgan, D. L. (2014b). Integrating qualitative and quantitative methods: A pragmatic approach. SAGE. https://doi.org/10.4135/9781544304533.n3 Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in the Schools, 13(1), 48–63. Pailthorpe, B. C. (2017). Emergent design. In J. Matthes (Ed.), The international encyclopedia of communication research methods (pp. 1–2). John Wiley. https://doi.org/10.1002/9781118901731. iecrm0081 Pallmann, P., Bedding, A. W., Choodari-Oskooei, B., Dimairo, M., Flight, L., Hampson, L. V., Holmes, J., Mander, A. P., Odondi, L., Sydes, M. R., Villar, S. S., Wason, J. M. S., Weir, C. J., Wheeler, G. M., Yap, C., & Jaki, T. (2018). Adaptive designs in clinical

EMBRACING EMERGENCE IN MIXED METHODS DESIGN

trials: Why use them, and how to run and report them. BMC Medicine, 16(1), 29. http://doi. org/10.1186/s12916-018-1017-7 Pluye, P., & Granikov, V. (2022). Combinations of practical integration strategies used in mixed methods information studies. In N. Ngulube (Ed.), Handbook of research on connecting research methods for information science research (pp. 1– 639). IGI Global. http://doi.org/10.4018/978-17998-8844-4.ch004 Pluye, P., & Hong, Q. N. (2023). Convergence and divergence in mixed methods research. In R. Tierney, F. Rizvi and K. Ercikan (Eds.), International encyclopedia of education (4th ed.), Volume 12 ‘Qualitative, Multimethod, and Mixed Methods Research’ (N. V. Ivankova, Ed.), pp. 1–16. Elsevier. Schoonenboom, J., & Johnson, R. B. (2017). How to construct a mixed methods research design. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(2), 107–131. http://doi.org/10.1007/ s11577-017-0454-1

113

Taylor, A., & Raykov, M. (2020). Towards critical and dialogical mixed methods research: Reflections on our journey. In B. Grummell & F. Finnegan (Eds.), Doing critical and creative research in adult education (pp. 127–137). Brill. http://doi.org/10.1163/97 89004420755_011 Tolich, M. (2016). How do emergent research questions confound mixed methods ethics? In M. Tolich (Ed.), Qualitative ethics in practice (1st ed., pp. 49–58). Routledge. Tom, A. (1996). Building collaborative research: Living the commitment to emergent design. International Journal of Qualitative Studies in Education, 9(3), 347–359. http://doi.org/10.1080/0951839960090309 Walker, C. (2015). Social constructionism and qualitative research. Journal of Theory Construction & Testing, 19(2), 37–38. Wright, H. (2009). Trusting the process: Using an emergent design to adult education. Educate: The Journal of Doctoral Research in Education, 9, 62–73.

8 The Methods-Inference Map: Visualizing the Interactions Between Methods and Inferences in Mixed Methods Research Judith Schoonenboom

INTRODUCTION In its simplest form, mixed methods research involves two acts of data collection and data analysis, commonly called “research strands”, one including qualitative data collection and data analysis, and one including quantitative data collection and data analysis (Bryman, 2008; Creswell & Plano Clark, 2018). Consequently, mixed methods researchers must develop their design at two levels. They must design each individual research strand such that its research question can be answered using the chosen sample, methods of data collection, and methods of data analysis. They must design the study as a whole such that its individual research strands together answer the overall research question and achieve the overall research goal. Thus, mixed methods researchers must attend to both their individual research strands and their study as a whole. This makes mixed methods research a complex endeavour (Poth, 2018). To get a better grasp of such complexity, mixed methods researchers frequently use visualizations to support combination and integration in all phases of their study (Shannon-Baker & Edwards, 2018). Combining or integrating the findings of individual research strands is supported by joint displays (Guetterman, Creswell et al., 2015; Guetterman, Fetters et al., 2015), visualizations that combine qualitative and quantitative elements (Creamer,

2020; Fetters, 2020; Plano Clark & Sanders, 2015; Wheeldon, 2010). Joint displays can also support combining and integrating research strands in data analysis (Bazeley, 2018; Fetters, 2020); interactive joint displays support the joint analysis of quantitative and qualitative data (Schoonenboom & Johnson, 2021; see also Chapter 25). For a discussion of using visuals to teach and learn about mixed methods research, see also Chapter 31. Visualizations can also support mixed methods design. For example, Maxwell’s (2013) interactive model of research design enables researchers to reflect the fit of their study’s main components. For a further discussion, see also Chapter 2. Other visualizations show the study’s design flow—that is, how the research strands follow after each other (Ivankova & Stick, 2007; Morse & Niehaus, 2009; Tashakkori & Teddlie, 2003). Furthermore, visualizations of a selection of elements for all separate research strands can be used to reflect the links between these elements (Bazeley, 2018; Miles & Huberman, 1994; Plano Clark & Sanders, 2015). Such visualizations have proven their worth; the Journal of Mixed Methods Research now requires submissions of an empirical study to include a joint display of its findings and a flow diagram of the data analysis and data collection procedures (Fetters & Molina-Azorin, 2019). Visualizations differ in what they can and cannot show about a research design. Some visualizations

115

THE METHODS-INFERENCE MAP

present a static overview of the study, whereas others display a study’s research flow. Some make a visual distinction between qualitative and quantitative components, whereas others do not. They also differ widely in what they show about the study’s structure. Finally, only some visualizations display integration points between the qualitative and quantitative components, such as the meta-inference, the overall conclusion at the end of a mixed methods study. No visualization displays all these elements. Consequently, researchers who switch from an overall view of their study to the details of their separate quantitative and qualitative research strands also have to switch visualizations. Can we avoid such switching between visualizations? Can we have one comprehensive visualization that shows an overview of the whole study, its structure, research flow, qualitative and quantitative components, and integration points? Such a visualization would provide a convincing summary of a mixed methods study in research proposals. More importantly, a comprehensive visualization would allow mixed methods researchers to zoom in and out from an overall view to the details of each research strand while designing their study individually, in a team or while analyzing mixed methods studies in teaching. This is the assignment I set myself, and the outcome is the methods-inference map, which I

will present in this chapter. I begin by discussing four different existing visualizations one by one, followed by a comparison of their strengths and limitations. Next, I present my methods-inference map, which builds on aspects of these existing visualizations, resulting in a detailed map of the various research strands, which shows the study’s design flow and how the meta-inference develops. Finally, I will sketch the implications of the methods-inference map for mixed methods research.

DESIGN VISUALIZATIONS WITH DIFFERENT PURPOSES Reflecting the Fit of Design Components (Maxwell, 2013) At the level of the whole study, researchers can use Maxwell’s (2013) interactive model of research design to inspect the fit between their design components. Maxwell emphasizes that researchers are not obliged to design their components in a specific order. Instead, their main task is to ensure that the design components fit each other at the end of the design process. An empirical example is presented in Figure 8.1. CONCEPTUAL FRAMEWORK

GOALS Improve adult learning in nontraditional settings. Bring adult development theory to empowerment curricula. Promote future academic career. RESEARCH QUESTIONS

METHODS

Own background in nontraditional education. Adult learning theory. Adult cognitive development theory. Literature on mediation and adult development. Own experiences as a patient.

What are the patients’ perceptions and practice of the cognitive skills taught? What did the patients learn, and how? How are the patients’ perceptions and practice related to their developmental level? What are the group leaders’ views of the curriculum and goals of the program? What is the cultural construction of stress in this program?

Interviews, both open-ended and developmental. Participant observation of program as patient. Program documents. Developmental analysis. Cultural analysis.

VALIDITY Triangulation of sources, methods, and theories. Search for discrepant evidence. Comparison with other programs in the literature.

Figure 8.1  Reflecting the fit of design components Source: Figure 1.3 A design map of Maria Broderick’s dissertation research. From Qualitative Research Design: An Interactive Approach (3rd ed., p. 9), by J. A. Maxwell (2013). Sage. Copyright 2013 by Sage. Reprinted with permission under the STM Guidelines.

116

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

In Figure 8.1, the study’s design components, are arranged in a square and connected by arrows. Researchers can compare the contents of the two boxes at both ends of each arrow and reflect on the fit between goals, theory, research questions, methods and validity at the level of the whole study.

Identifying Links Between Specific Design Components An important design question for each separate research strand is how to answer its research question.

Bazeley’s (2018) model in Table 8.1 visualizes the links between a specific research question, its theoretical basis, the data sources used to answer it, and the methods of data analysis. Its underlying principle is “Create one row for each separate research question”, a principle that, in an extended form, will play an important role in my methodsinference map. Researchers can use Table 8.1 to reflect, for example, whether a specific data source is indeed fit to answer the specific research question to which it is attached. Bazeley’s model is not limited to specific research components. Another model, for instance, focuses on connecting data sources, information gained, and understanding

Table 8.1  Linking research questions, theoretical basis, data sources and methods of data analysis Specific research questions

Theoretical/conceptual basis

What does wellbeing mean for older women?

Goertz – theorising concepts Free-listing, pile-sorts, photo Capability – agency elicitation, drawings approach to life well Interviews lived

How does participation in OWN programmes impact on health and wellbeing?

What motivates and maintains attendance?

What is the relationship between physical health and general wellbeing?

Data sources

Data analysis

Cultural domain statistics Concept analysis – identify dimensions Critical realist analysis – attributes, structure, environment, agency Active ageing/healthy ageing Annual (linked) surveys – Statistical analyses: theories and research variables include use of descriptive stats, health services, health repeated measures, and wellbeing indicators, comparison with national physical and social data, relationships activity between age, attendance, QL comments on aspects of activity levels and participation outcomes Observations and MM – relate comments to attendance statistics indicators of participation Case studies of participants and outcomes; build case profiles from multiple sources. Within and cross-case analysis Social connection as Longevity/regularity/attrition Descriptive statistics motivator for physical in attendance stats; Connect attendance stats activity classes selected and survey responses Annual surveys (QT + QL) with reasons for coming and stated benefits of coming Adaptability Surveys Statistical – relationships Life satisfaction set-point Interviews – aspects of between H and WB theory health and wellbeing, variables Self-determination theory incl. agency, autonomy, Theorising relationship competence (grounded theory; CR – identifying mechanisms in relationship)

Source: Table 2.1 Questions and methods for the Wellbeing Project. From Integrating Analyses in Mixed Methods Research (p. 34), by P. Bazeley (2018). Sage. Copyright 2018 by Patricia Bazeley. Reprinted with permission from the publisher under the STM Guidelines and from the author.

THE METHODS-INFERENCE MAP

gained (Bazeley, 2018, p. 35; for similar visualizations, see Miles & Huberman, 1994; Plano Clark & Sanders, 2015). Researchers can productively combine Bazeley’s and Maxwell’s models in one study. They can use Maxwell’s model to design their components at the overall level and Bazeley’s model to reflect connections at a detailed level: Bazeley’s model splits Maxwell’s “Research questions” into “Specific research questions”, his “Conceptual Framework” into separate “Theoretical/conceptual bases”, and his “Methods” into specific “Data sources” and “Data analyses”.

Visualizing the Design Flow The previous two visualization models focused on the fit of design elements at the level of the whole

117

study (Maxwell) or the separate research questions (Bazeley). Other visualization models focus on the design flow, on how design components unfold in time (Creamer, 2018; Morse & Niehaus, 2009; Plano Clark & Sanders, 2015; Tashakkori et al., 2021), and especially on data collection and data analysis (Bazeley, 2018; Ivankova & Stick, 2007). Visualizing the design flow is helpful in complex research such as mixed methods research because it makes visible the various design decisions that are made during the study—for example, as researchers move from one research strand to the next, thereby enabling reflection on these decisions. Teddlie and Tashakkori’s (2009) model in Figure 8.2 is an abstract model of the design flow of a study. In this explanatory sequential mixed methods study (Creswell & Plano Clark, 2018), a quantitative research strand, indicated by rectangular concept boxes, is followed by a

Figure 8.2  Design flow of an explanatory sequential design Source: Figure 7.5 Graphic illustration of sequential mixed designs. From Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences (p. 154), by C. B. Teddlie and A. Tashakkori (2009). Copyright 2009 by Sage. Reprinted with permission under the STM Guidelines.

118

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

qualitative research strand, indicated by elliptical concept boxes. Each research strand consists of the same stages, and the corresponding stages of each research strand are displayed at the same height. At the end of the study, inferences derived from both research strands are

integrated into the meta-inference, the study’s overall conclusion. Ivankova et  al.’s (2006) model in Figure 8.3 shows the design flow of a specific explanatory sequential mixed methods study; it shows how the concrete research activities are performed

Figure 8.3  Design flow of an explanatory sequential mixed methods study Source: Figure 1 Visual model for mixed-methods sequential explanatory design procedures. From Using Mixed-Methods Sequential Explanatory Design: From Theory to Practice, by N. V. Ivankova, J. W. Creswell and S. L. Stick (2006), Field Methods, 18(1), p. 16. https://doi.org/10.1177/1525822x05282260. Reprinted with permission of Sage Publications Inc., from Field Methods, 18(1) 2006. Permission conveyed through Copyright Clearance Center, Inc.

119

THE METHODS-INFERENCE MAP

one after the other. The rectangles and ellipses in Figure 8.3 have different meanings from those in Figure 8.2. Consecutive rectangles indicate components that belong to one research strand and are performed one after the other; ellipses indicate points where the qualitative and quantitative research strands are brought together and integrated. The study in Figure 8.2 has two integration points: after finishing the quantitative research strand, the quantitative data are used to draw a sample and develop the interview questions for the qualitative research strand. At the end of the study, findings of the quantitative and qualitative research strands are integrated into one meta-inference.

Strengths and Weaknesses As stated in the Introduction, visualizations differ in what they can and cannot show about a research design. Table 8.2 summarizes the discussion of the four visualizations, adding what each visualization cannot show. No visualization shows all design aspects. Maxwell’s model does not provide information about a study’s flow, the structure of separate research strands, or forms of integration. Bazeley’s model does not contain information about the study as a whole, its design flow, or its metainference. Teddlie and Tashakkori’s model does not contain content, and thus does not allow for the content comparisons. Ivankova et al.’s model gives little information about a study’s structure. Also, it does not provide an overview of the whole study, but only of the defining elements of a mixed

methods study: quantitative data collection and data analysis, qualitative data collection and data analysis, and integration. However, although no visualization provides all design aspects, together they do. Each aspect is displayed in at least one of the visualizations. This opens up the possibility of creating a comprehensive visualization: the methods-inference map.

THE METHODS-INFERENCE MAP My Empirical Example: McCrudden and McTigue (2019) My methods-inference map aims to combine the strengths of the four visualizations and enable researchers to zoom in and out between the level of the whole study and the individual research strands. In this chapter, my methods-inference map is applied to an empirical example—namely, McCrudden and McTigue (2019), a methodological article that illustrates how in a mixed methods study, integration can be achieved in an explanatory sequential design at the methods level, through the sampling frame and through the development of the interview protocol with a methodological joint display, and at the interpretation and reporting level through narrative and the use of a results joint display. (p. 318)

The example’s topic is belief bias in reasoning. Someone with a belief bias evaluates the

Table 8.2  Design elements shown and not shown by each of the four visualizations Design aspect

Maxwell (2013)

Teddlie and Tashakkori (2009)

Ivankova et al. (2006)

Bazeley (2018)

Overview of the study Research elements’ contents Visual distinction between qualitative and quantitative elements Design flow Study’s research strand structure Meta-inference Other forms of integration Comparison of contents at the overall level Comparison of contents at a detailed level

+ + -

+ +

++ -

+ -

+ -

+ + + 0 -

+ 0 + + -

+

Note: + = good; +- = good, but…; 0 = fair; - = absent

120

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

information in an argument on its consistency with their beliefs rather than on the quality of the evidence, which favours (“bias”) belief-consistent arguments over belief-inconsistent arguments (see also McCrudden & Barnes, 2016). In this study among adolescents in one all-male suburban public secondary school in New Zealand, McCrudden and McTigue (2019) answered the research question, “How do adolescents evaluate belief-consistent and belief-inconsistent arguments that have equally compelling justifications?” In the quantitative phase, participants rated the strength of arguments about climate change that were supported by plausible fictional data that were consistent or inconsistent with their beliefs. [ … ] In the follow-up qualitative phase, a purpose-

fully sampled subset of the participants from the quantitative phase were interviewed to gain insights into the reasoning behind their evaluation of the arguments. (McCrudden & McTigue, 2019, pp. 384–385)

Overall, the students showed belief bias. There was, however, one crucial difference. In their argumentation, students with low belief bias focused on the quantity of evidence and applied the same evaluation criteria independently of whether the arguments were belief-consistent. Conversely, students with high belief bias “applied different standards of evaluation based on whether the arguments were belief-consistent rather than on the evidence presented in the arguments” (McCrudden & McTigue, 2019, p. 394).

Figure 8.4  A methods-inference map of McCrudden and McTigue (2019) Source: Created by the author using Inspiration (Inspiration Software, 2021).

THE METHODS-INFERENCE MAP

The Methods-inference Map: An Overview A methods-inference map of McCrudden and McTigue (2019) is provided in Figure 8.4. The methods-inference map resembles a detailed variant of Teddlie and Tashakkori’s model: like Figure 8.2, the methods-inference map displays the design flow through vertically oriented research strands, which are placed next to each other, representing their chronological order. Like Figure 8.2, a quantitative Research Strand 1 (represented by rectangular concept boxes on the left-hand side) is followed by a qualitative Research Strand 2 (represented by elliptical concept boxes on the right-hand side). Compared with Teddlie and Tashakkori’s model, Figure 8.4 contains many more details. First, its concept boxes contain the contents of McCrudden’s study. In addition, Figure 8.4 contains four instead of two research strands. Based on its two different research questions and subsequently different quantitative data and data analysis, Research Strand 1 has been split into Research Strand 1a and Research Strand 1b. Based on its two different populations and hence different samples, qualitative Research Strand 2 has been split into Research Strand 2a and Research Strand 2b. Finally, the one upward arrow of Teddlie and Tashakkori’s model between Research Strand 1 and Research Strand 2 has been split into nine different arrows, each representing a different relationship between Research Strand 1 and Research Strand 2. Furthermore, Figure 8.4’s four detailed research strands are variants of Bazeley’s Table 8.1. Like Table 8.1, they show for each specific research question which methods of data collection and methods of data analysis are used to answer that question. Bazeley’s “theoretical basis” is not included, but it could be if needed. Finally,

121

Figure 8.4 contains elements of Maxwell’s and Ivankova et al.’s models, which will be discussed in more detail below.

THE METHODS-INFERENCE MAP IN DETAIL Design Components at the Level of the Whole Study Figure 8.5 contains four design elements at the level of the whole study, allowing researchers to reflect the fit between them. The figure distinguishes between an immediate goal and a distant goal (Schoonenboom, 2018). The immediate goal is what researchers aim to achieve in their study, whereas the distant goal is the furtherreaching goal they wish to contribute to (notice the word “contribute” in Figure 8.5’s distant goal’s text). At least three cases of fit can be identified: • The immediate goal fits the distant goal: if we understand how adolescents evaluate beliefrelevant arguments about climate change (immediate goal), we contribute to developing scientific reasoning and minimizing the influence of belief bias (distant goal). More concrete: if we understand how and when belief bias emerges, we can intervene with the aim that belief bias disappears, and sound scientific reasoning takes its place. • The distant goal, the immediate goal, and the overall research question all refer to “belief bias among adolescents” and thus fit this phenomenon of interest.

Figure 8.5  Design components at the level of the whole study in Figure 8.4 Source: Created by the author using Inspiration (Inspiration Software, 2021).

122

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

• The overall research question fits the immediate goal. Suppose we answer the research question, “How do adolescents evaluate belief-consistent and belief-inconsistent arguments that have equally compelling justifications?” In this case, we understand how adolescents evaluate beliefrelevant arguments about climate change (immediate goal). Figure 8.5 contains two of Maxwell’s elements: goals and research questions. Like Maxwell’s model, it can be used to reflect these components’ fit. However, it also contains elements of a design flow. The idea is that the phenomenon is determined first, followed by the distant goal, then the intermediate goal, and finally the research question. Reflecting the fit is thus seen as feedback loops, returning to an earlier stage of the process.

Maxwell’s theoretical framework or validity are not included, but they could be included as a separate node at the overall or research strand level. Maxwell’s methods are not included at the overall study level but at the research strand level, discussed below.

Comparing Design Components Across Levels Researchers can use Figure 8.4 to reflect across the whole study and the individual research strand level. For instance, they can ask whether the combined answers to the research questions of the research strands answer the overall research question. Such reflection is not included in the previous visualizations.

Figure 8.6  Research questions at the level of the whole study and the level of the research strands in Figure 8.4 Source: Created by the author using Inspiration (Inspiration Software, 2021).

Figure 8.6 shows that Research Questions 2a and 2b are used to answer the overall research question. Different answers are provided for the two different populations: adolescents with low belief bias and those with high belief bias. Research Questions 1a and 1b do not play a role in answering the overall research question. Research Question 1b’s role is to distinguish between adolescents with high belief bias and low belief bias, a distinction that is transferred to Research Questions 2a and 2b. Research Question 1a is used to establish an overall effect, which is later abandoned in favour of different effects for different populations. Elsewhere, I have called such research questions with a temporary role “intermediate research questions” (Schoonenboom, 2016).

Research Strand 1 In Figure 8.7, Teddlie and Tashakkori’s first, quantitative research strand has been split into two

substrands, Research Strands 1a and 1b. Research Strands 1a and 1b have different research questions, which are answered through different types of data analysis. This leads to different results and, finally, different conclusions: “Adolescents show belief bias” for Research Strand 1a and “Some students show high belief bias, whereas others show low belief bias” for Research Strand 1b. Within each substrand, research components are connected, as in Bazeley’s Table 8.1. This could be formulated as a general principle: include separate research strands for each component that will lead to a separate conclusion, or, more succinctly, “Use one research strand to develop one conclusion”. In line with this principle, research strands are split whenever a researcher includes more than one research question, population, method of data analysis, or data source. Research Strands 1a and 1b are viewed as substrands of Research Strand 1 rather than entirely independent research strands because they share several components: the population, sample, and method of data collection. In Figure 8.7, each shared component is displayed as

THE METHODS-INFERENCE MAP

123

Figure 8.7  Research strand 1 in Figure 8.4 Source: Created by the author using Inspiration (Inspiration Software, 2021).

one concept box, which receives and sends arrows to and from both substrands. Within Research Strand 1, there is a fit between its components’ research question, data collection, data, and data analysis. Arguments on the acceptance of human-induced climate change (data collection) can be classified as belief-consistent or belief-inconsistent (research question), showing whether belief-consistent arguments are rated differently from belief-inconsistent arguments (data analysis). In addition, each individual substrand displays a fit between data analysis and results. Repeated measures ANOVA can be used to establish a main effect (Research Strand 1a), whereas

descriptive statistics can be used to reveal differences between students (Research Strand 1b). A less-than-perfect fit exists between the research question and the population. Whereas the research question refers to “adolescents” generally, the researched population were students at suburban all-male public secondary schools near the coast in New Zealand. The results cannot easily be generalized to students from different areas (urban, rural, or inland), genders (female), countries (outside New Zealand), or school types (private schools), nor to adolescents with different daytime activities (working, unemployed, studying).

124

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Research Strand 2 Like Research Strands 1a and 1b, Research Strands 2a and 2b have different research questions, in this case because each research question refers to a different population, which also leads to different samples. Research Strand 2a studies students with low belief bias, whereas Research Strand 2b studies students with high belief bias. The conclusion for each population is different; hence, Research Strand 2 has been split into two substrands, according to the principle, “Use one research strand to develop one conclusion”. Research Strands 2a and 2b share some other components: a large part of their research question, the method of data collection, the data, and the method of data analysis. Each substrand shows a fit between the research questions, population, sampling, and methods of data collection and data analysis. Most elements of each research strand refer to its population, “students with low belief bias” and “students with high belief bias” respectively. In line with the open “how” question, the data collection and data analysis methods of Research Strand 2 are open. The sample size is small to allow for in-depth data collection.

•

•

•

Connections Between the Research Strands In addition to the meta-inference, qualitative and quantitative research strands in a mixed methods study are connected or integrated at various integration points (see, for instance, the integration point ellipse in Figure 8.3). These integration points reflect important mixed methods design decisions—namely, how to use the former research strand to create the latter (Fetters et al., 2013). As in most mixed methods studies, McCrudden and McTigue’s (2019) second research strand builds on the first research strand in various ways (Schoonenboom et  al., 2017). Figure 8.4 contains nine arrows from Research Strand 1 to Research Strand 2 (for a different visualization of such connections, see Plano Clark & Sanders, 2015, p. 192): • Conclusion of Research Strand 1b (henceforth: Conclusion 1b) to Populations of Research Strands 2a and 2b (henceforth: Populations 2a and 2b). Research Strand 1 revealed that some students showed high belief bias, whereas others showed low belief bias. This conclusion divides the population of Research Strand 2 into

•

•

two different populations. Research Strand 2 contains two substrands, one for the population of students with high belief bias and one for the population of students with low belief bias. Conclusion 1b to Research Question 2. Conclusion 1b is incorporated in Research Strand 2’s research question: “How do adolescents with high and low belief bias evaluate beliefconsistent and belief-inconsistent arguments that have equally compelling justification?” It is important to note here that only the conclusion of Research Strand 1b, “Some students show high belief bias, whereas others show low belief bias”, is connected to Research Strand 2 in various ways. Conclusion 1a, “Adolescents show belief bias”, is irrelevant for designing Research Strand 2. Conclusion 1a is also irrelevant for developing the meta-inference, which will be discussed in the next section. Sample 1 to Samples 2a and 2b. One arrow from Sample 1 to Samples 2a and 2b indicates that Samples 2a and 2b are subsamples of Sample 1. Data 1 to Samples 2a and 2b. The individual belief bias scores of Research Strand 1 are used as the basis for Sample 2: one arrow leads to a subsample of four students with high belief bias, whereas one other arrow leads to a subsample of four students with low belief bias. Rating scores 1 to Data collection 2. The quantitative data of Research Strand 1 are used to develop the interview protocol of Research Strand 2. Conclusion 1b to Conclusions 2a and 2b. One arrow signals that Conclusion 1b is further developed into Conclusions 2a and 2b. After that, Conclusions 2a and 2b are combined in the meta-inference. The development of the metainference is the topic of the next section.

Teddlie and Tashakkori’s one upward arrow between Research Strand 1 and Research Strand 2 (Figure 8.2) has been split into nine different arrows, each representing a different relation between Research Strand 1 and Research Strand 2.

Developing the Meta-Inference Researchers use methods ultimately to develop conclusions, inferences (see also Cartwright &

THE METHODS-INFERENCE MAP

125

Figure 8.8  Research strand 2 in Figure 8.4 Source: Created by the author using Inspiration (Inspiration Software, 2021).

Hardie, 2012). The methods-inference map received its name because it shows how methods are used to draw inferences, which are further developed into one overall meta-inference at the end of a study (Schoonenboom, 2022). Although each individual research strand leads to one conclusion, a mixed methods study does not consist of various separate, unrelated conclusions. Instead, several conclusions build on each other, and ever more sophisticated conclusions are developed, culminating in the meta-inference. This development process is visible at the bottom of Figure 8.4, repeated here as Figure 8.9: • Conclusions 2a and 2b build on Conclusion 1b because each refers to one population identified

in Conclusion 1b, “Students with low belief bias” and “Students with high belief bias”, respectively. • The meta-inference builds on Conclusions 2a and 2b. In the meta-inference, the researchers compare and connect Conclusions 2a and 2b and note that they are different: “Different from students with low belief bias, students with high belief bias focused on the quantity of evidence only when the arguments were beliefinconsistent”. • Conclusion 1a, “Adolescents show belief bias,” does not play a role in developing the metainference. The meta-inference is solely based on the further development of Conclusion 1b, “Some students show high belief bias, whereas others

126

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 8.9  Developing the meta-inference in Figure 8.4 Source: Created by the author using Inspiration (Inspiration Software, 2021).

show low belief bias”. Conclusion 1a does not play a role in developing the meta-inference or designing Research Strand 2, as indicated by the lack of outgoing arrows from Conclusion 1a in Figure 8.4. The meta-inference development process in the methods-inference map differs in important ways from Teddlie and Tashakkori’s and Ivankova et  al.’s models. In a methods-inference map, the meta-inference develops gradually and not by integrating conclusions of the individual research strands at the end of a study. Throughout a study, we have a conclusion that is the meta-inference of that moment. Being a further development, the meta-inference also often does not integrate the conclusions of Research Strand 1 and Research Strand 2, but, for example, of Research Strand 2a and Research Strand 2b, as shown in Figure 8.9 (for a detailed description of this process, see Schoonenboom, 2022).

CONCLUSIONS This chapter started with the premise that mixed methods research design is complex and that mixed methods researchers must maintain an overview of their study while at the same time scrutinizing their separate qualitative and quantitative research strands. From this premise, it proved problematic that researchers must use different visualizations when switching from an overall view of their study to the details of their quantitative and qualitative research strands because current visualizations do not provide all relevant information. Next, four visualizations were examined in detail, investigating their strengths and asking whether these strengths could be combined in one visualization that would allow for such zooming in and out.

The resulting methods-inference map can now be summarized as a visualization with the structure of Teddlie and Tashakkori’s model, with its vertically oriented research strands placed next to each other, thereby enabling a visual overview of very complex research designs. Adding the contents of a specific study to this structure shows its design flow, similar to Ivankova et  al.’s model, while at the same time enabling Bazeley’s comparisons between the elements of each research strand and Maxwell’s comparison of research components at the level of the whole study. A methods-inference obtains its details from the principle, extended from Bazeley’s model: “Use one research strand to develop one conclusion”. This principle results in more research strands than are distinguished in Teddlie and Tashakkori’s and Ivankova et al.’s models, as became visible in McCrudden and McTigue’s (2019), where both the quantitative and the qualitative research strands were split into two strands: one for each research question and each population. The methods-inference map extends the possibilities of reflecting fit and scrutinizing flow. By covering the level of the study as a whole, the individual research strands, and their connections at a detailed level, the possibilities for reflecting the fit between components provided by Maxwell’s and Bazeley’s models are combined. In addition, we can reflect the fit between research strand components and overall components of the same type. We can see, for example, whether the research questions of the different research strands together cover the overall research question (Figure 8.6). Each anticipated conclusion has its own research strand, and thus, a methods-inference map promotes reflection at different levels. Looking at the visualization vertically, we can see how the conclusion of each individual research strand is developed using methods. Looking at the visualization horizontally, we can obtain an overview of the whole study. For instance, we can see

THE METHODS-INFERENCE MAP

how the conclusions develop into a meta-inference (Figure 8.9). We can see the unique contribution of each research strand’s conclusion to the development of the meta-inference. We can also see which conclusions are not involved in developing the meta-inference. A methods-inference map is helpful in all situations where reflection on a study as a whole and on its details is essential—for example, when working in a team, submitting proposals, or analyzing mixed methods studies for teaching purposes. The methods-inference map provides important insights into how the meta-inference develops throughout the study. It reveals what is hidden in McCrudden and McTigue (2019): The first group conclusion, “Students show belief bias”, is not used to develop the meta-inference, indicated by a lack of arrows from this conclusion. Instead, the meta-inference is developed from the different conclusions for the different populations of students with high versus low belief bias in Research Strands 2a and 2b. The principle “Use one research strand to develop one conclusion” has various implications for mixed methods research. Researchers should include their anticipated or possible outcomes in their designs (Bazeley, 2018) because their number determines the number of research strands. In addition, researchers must include one research strand for each population, allowing them to explore differences between populations. As we saw in McCrudden and McTigue (2019), exploring such differences makes sense, and this is the case for many mixed methods studies (Schoonenboom, 2019). Thus, drawing a methods-inference map makes sense. It enables us to switch between our overall study and its separate research strands. Its many details enable us to see the many connections between research strands and give us a more accurate picture of how the meta-inference develops. I hope that mixed methods researchers will find inspiration from this chapter to develop their own methods-inference maps that fit their needs.

WHAT TO READ NEXT Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage.

This classic textbook contains many visualizations that are useful for mixed methods design. Recently, two further editions have been published

127

by Saldana (Miles et  al., 2020) under the same title, but with substantial differences. Plano Clark, V. L., & Sanders, K. (2015). The use of visual displays in mixed methods: Research strategies for effectively integrating the quantitative and qualitative components of a study. In M. T. McCrudden, G. Schraw, & C. W. Buckendahl (Eds.), Use of visual displays in research and testing (pp. 177–206). Information Age Publishing.

This article brings together the use of visualizations for various design-related goals, including instrument development and integration. Shannon-Baker, P., & Edwards, C. (2018). The affordances and challenges to incorporating visual methods in mixed methods research. American Behavioral Scientist, 62, 935–955. https://doi. org/10.1177/0002764218

Those interested in how mixed methods researchers can use visual methods or visualizations as methods, should read this article and other work by Peggy Shannon-Baker.

REFERENCES Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage. Bryman, A. (2008). Why do researchers integrate/ combine/mesh/blend/mix/merge/fuse quantitative and qualitative research? In M. M. Bergman (Ed.), Advances in mixed methods research (pp. 87– 100). Sage. Cartwright, N., & Hardie, J. (2012). Evidence-based policy: A practical guide to doing it better. Oxford University Press. Creamer, E. G. (2018). An introduction to fully integrated mixed methods research. Sage. Creamer, E. G. (2020). Visualizing dynamic fully integrated mixed method designs. International Journal of Multiple Research Approaches, 12(1), 65–77. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Fetters, M. D. (2020). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Sage. Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs— principles and practices. Health Services Research, 48(6.2), 2134–2156. https://doi.org/10.1111/ 1475-6773.12117

128

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Fetters, M. D., & Molina-Azorin, J. F. (2019). A checklist of mixed methods elements in a submission for advancing the methodology of mixed methods research. Journal of Mixed Methods Research, 13(4), 414–423. https://doi.org/10.1177/155868 9819875832 Guetterman, T., Creswell, J. W., & Kuckartz, U. (2015). Using joint displays and MAXQDA software to represent the results of mixed methods research. In M. T. McCrudden, G. Schraw, & C. W. Buckendahl (Eds.), Use of visual displays in research and testing (pp. 145–175). Information Age Publishing. Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm. 1865 Inspiration Software. (2021). Inspiration (Version 10) [Computer software]. TechEd Marketing. www. inspiration-at.com/inspiration-10/ Ivankova, N. V., Creswell, J. W., & Stick, S. L. (2006). Using mixed-methods sequential explanatory design: From theory to practice. Field Methods, 18(1), 3-20. https://doi.org/10.1177/1525822x05282260 Ivankova, N. V., & Stick, S. L. (2007). Students’ persistence in a distributed doctoral program in educational leadership in higher education: A mixed methods study. Research in Higher Education, 48(1), 93–135. https://doi.org/10.1007/s11162006-9025-4 Maxwell, J. A. (2013). Qualitative research design: An interactive approach (3rd ed.). Sage. McCrudden, M. T., & Barnes, A. (2016). Differences in student reasoning about belief-relevant arguments: A mixed methods study. Metacognition and Learning, 11(3), 275–303. https://doi. org/10.1007/s11409-015-9148-0 McCrudden, M. T., & McTigue, E. M. (2019). Implementing integration in an explanatory sequential mixed methods study of belief bias about climate change with high school students. Journal of Mixed Methods Research, 13(3), 381–400. https:// doi.org/10.1177/1558689818762576 Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage. Miles, M. B., Huberman, A. M., & Saldana, J. (2020). Qualitative data analysis: A methods sourcebook (4th ed.). Sage. Morse, J. M., & Niehaus, L. (2009). Mixed method design: Principles and procedures. Left Coast Press. Plano Clark, V. L., & Sanders, K. (2015). The use of visual displays in mixed methods: Research strategies for effectively integrating the quantitative and

qualitative components of a study. In M. T. McCrudden, G. Schraw, & C. W. Buckendahl (Eds.), Use of visual displays in research and testing (pp. 177–206). Information Age Publishing. Poth, C. N. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Schoonenboom, J. (2016). The multilevel mixed intact group analysis: A mixed method to seek, detect, describe, and explain differences among intact groups. Journal of Mixed Methods Research, 10(2), 129–146. https://doi.org/10.1177/ 1558689814536283 Schoonenboom, J. (2018). Designing mixed methods research by mixing and merging methodologies: A 13-step model. American Behavioral Scientist, 62(7), 998–1015. https://doi.org/10.1177/ 0002764218772674 Schoonenboom, J. (2019). Develop your case! How controversial cases, subcases, and moderated cases can guide you through mixed methods data analysis. Frontiers in Psychology, 10(1369). https:// doi.org/10.3389/fpsyg.2019.01369 Schoonenboom, J. (2022). Developing the metainference in mixed methods research through successive integration of claims. In Hitchcock, J. H. and Onwuegbuzie, A. J. (eds.) The Routledge handbook for advancing integration in mixed methods research (pp. 55–70). Routledge. https:// doi.org/10.4324/9780429432828-6 Schoonenboom, J., & Johnson, R. B. (2021). The case comparison table: A joint display for constructing and sorting simple tables as mixed analysis. In A. J. Onwuegbuzie & R. B. Johnson (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 277–288). Routledge. https://doi.org/ 10.4324/9780203729434-24 Schoonenboom, J., Johnson, R. B., & Froehlich, D. E. (2017). Combining multiple purposes of mixing within a mixed methods research design. International Journal of Multiple Research Approaches, 10(1), 271–282. https://doi.org/ 10.29034/ijmra.v10n1a17 Shannon-Baker, P., & Edwards, C. (2018). The affordances and challenges to incorporating visual methods in mixed methods research. American Behavioral Scientist, 62(7), 935–955. https://doi. org/10.1177/0002764218772671 Tashakkori, A., & Teddlie, C. (2003). The past and future of mixed methods research: From data triangulation to mixed model designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 671–701). Sage. Tashakkori, A. M., Johnson, R. B., & Teddlie, C. B. (2021). Foundations of mixed methods research (2nd revised ed.). Sage.

THE METHODS-INFERENCE MAP

Teddlie, C. B., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage.

129

Wheeldon, J. (2010). Mapping mixed methods research: Methods, measures, and meaning. Journal of Mixed Methods Research, 4(2), 87–102. https://doi.org/10.1177/1558689809358755

9 Towards Sampling Designs that are Transparent, Rigorous, Ethical and Equitable (TREE): Using a Tree Metaphor as a Sampling Meta-Framework in Mixed Methods Research Julie A. Corrigan and Anthony J. Onwuegbuzie

INTRODUCTION The process of sampling likely represents the most important step of the research process— whether the study is a qualitative research study, a quantitative research study or a mixed methods research study. The importance of sampling stems from the fact that it does not matter how appropriate the research question(s), the research design, the instrument(s), and data analysis approaches are, if the selected sample(s) is inappropriate—for example, due to an inappropriate sampling design (e.g., type of sampling scheme[s] [i.e., purposive vs. random], sampling scheme[s] [e.g., simple random sampling, stratified random sampling, convenience sampling, maximum variation sampling; cf. Onwuegbuzie & Collins, 2007], sample size, subsample size[s], group size[s] per approach, number of observational units per participant)—then, the ensuing findings and interpretations will be untrustworthy at best and misleading at worst. Indeed, it could be argued that sampling is an even greater issue for mixed

methods research studies than for monomethod research studies because of their added complexity, resulting from the use of multiple samples. Thus, it is particularly surprising and troubling that there is a dearth of methodological guidance in the area of mixed methods sampling (Corrigan & Onwuegbuzie, in press; Hense, 2017). With a lack of guidance in sampling in the literature, it is no wonder that many, if not most, mixed methods research studies include mixed methods sampling designs that are problematic (see, for example, Onwuegbuzie & Corrigan, 2021)—that is, these sampling designs lack one or more of the following important attributes: transparency, rigour, equitableness, and ethicality. Broadly speaking: • Transparency means that mixed methods researchers should communicate clearly their sampling decisions in such a way that another researcher could replicate them, but also in a way in which they are transparent about their subjective positioning (i.e., positionality; Peshkin,

USING A TREE METAPHOR AS A SAMPLING META-FRAMEWORK

1988) and how that affects their sampling decisions and subsequent research findings. Further, transparency refers to clarifying the logic underpinning the sampling process (i.e., sampling logic), as well as any bias associated with their sampling decisions. • Rigour refers to making sampling decisions in a way that leads to findings and interpretations that are trustworthy. In the parlance of quantitative research, trustworthiness is referred to as validity and reliability (Johnson & Christensen, 2020), whereas in the language of qualitative research, this is referred to as (a) truth-value (credibility); (b) applicability (transferability); (c) consistency (dependability); and (d) neutrality (confirmability) (cf. Lincoln & Guba, 1985). • Equitableness is concerned with “the removal of systemic barriers and biases enabling all individuals to have equal opportunity to access and benefit from the [research] [… ]. To achieve this, all individuals who participate in the research ecosystem must develop a strong understanding of the systemic barriers faced by individuals from underrepresented groups (e.g., women, persons with disabilities, Indigenous Peoples, racialized minorities, individuals from the LGBTQ2+ community) and put in place impactful measures to address these barriers” (Canada Research Coordinating Committee, 2021). In sampling, this could involve ensuring that the population is accurately represented in the research study, in as far as is feasible. In other words, whether consciously or unconsciously, at times, researchers fail to represent certain segments of a population, whether simply for convenience or for more nefarious reasons. This could also involve naming the groups who hold the power—or are being harmed or privileged—via sampling decisions (to paraphrase the work of Randall and colleagues [2021] around justice-oriented approaches to assessment). For a further discussion of sampling from ethnically and racially minoritized groups in mixed methods research, see Chapter 25. • Ethicality refers to “protecting others, minimizing harm and increasing the sum of good” (Israel & Hay, 2006, p. 2). In sampling, this is just as much about whom/what we exclude as whom/what we include. For a more in-depth discussion of the TREE principles, we refer you to Corrigan and Onwuegbuzie (in press).

131

THE TREE SAMPLING META-FRAMEWORK As explained by Bazeley and Kemp (2012): Metaphors create images that facilitate understanding, communication, and remembering through using something familiar—such as eating, building, or agriculture—to explain or describe something new or more difficult to comprehend (Bonner & Greenwood, 2005). Metaphors often convey more than the literal meaning: Their implied meanings create new realities for recipients by affecting their perceptions and thus their actions (Krippendorff, 1993)—they are “a device of representation through which new meaning can be learned” (Coffey & Atkinson, 1996, p. 85). (pp. 56–57)

In this section, via the use of a metaphor, we propose a meta-framework to help researchers make sampling decisions that are transparent, rigorous, ethical, and equitable (TREE). Specifically, we capture our meta-framework via the metaphor of a tree. We illustrate how our tree meta-framework yields sampling designs that are TREE (i.e., tree à TREE). This tree approach has its roots in critical dialectical pluralism—a mixed methods research philosophy originally developed by Onwuegbuzie and Frels (2013) and expanded by Onwuegbuzie et al. (in press)—which represents a process philosophy and a communication theory that emphasizes procedural justice, process justice, and philosophical justice, and promotes both universalistic theoretical knowledge and local practical knowledge (see also Onwuegbuzie & Corrigan, 2021). In particular, we argue that sampling designs that are conceptualized, planned, and implemented using our tree meta-framework have a greater potential to have all four of the following TREE characteristics: • Transparency: By actually creating a tree-like figure (i.e., Figure 9.1) and including it in their final reports, mixed methods researchers will be in a position to communicate clearly their sampling logic, their sampling decisions, and the sampling process(es) involved. • Rigour: Via elements associated with the various parts of the tree, mixed methods researchers will be in a position to develop a sampling logic that would yield a rigorous sampling design that is “substantiated by coherence and connection among the constituent parts. The separate parts … [would] … fit together and work together to enable–from the perspective of a given inquiry approach—defensible data gathering, analysis, and interpretation’’ (Greene, 2006, p. 93).

132

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 9.1  Using a tree metaphor as a sampling meta-framework for enhancing representation in mixed methods research Adapted from Crossover displays for mixed methods sampling designs that are Transparent, Rigorous, Ethical, and Equitable (TREE) by J. A. Corrigan and A. J. Onwuegbuzie, 2022, unpublished manuscript, Concordia University, Montreal, Quebec, Canada, p. 1. Copyright 2022 by J. A. Corrigan and A. J. Onwuegbuzie.

In turn, this rigorous sampling design would yield findings and interpretations that are trustworthy and meaningful. • Equitableness: By enabling the tree approach to be driven by critical dialectical pluralism, mixed methods researchers will be in a position to develop a sampling design that is characterized by equity and fairness. • Ethicality: By enabling the tree approach to be driven by critical dialetical pluralism, mixed methods researchers will be in a position to develop

a sampling design that is not just ethical but, as described by Corrigan and Onwuegbuzie (in press), is meta-ethical, which implies adherence to virtue ethics (i.e., referring to the character of the mixed methods researcher as the impetus for ethical sampling behaviour, as opposed to merely focusing on rules) and pragmatic ethics (i.e., using the standards established by communities under the assumption that communities are progressing morally in line with the progression of scientific knowledge).

USING A TREE METAPHOR AS A SAMPLING META-FRAMEWORK

In what follows, we provide a street map to planning and implementing mixed methods sampling designs, wherein we offer practical guidance for mixed methods researchers. In our tree sampling meta-framework (Figure 9.2), we first consider the roots of the tree, which are our paradigmatic assumptions (e.g., one of the 14 mixed methods research philosophies – e.g., critical dialetical pluralism; cf. Onwuegbuzie

133

& Corrigan, 2021). Next, we consider the trunk of the tree, which are the foundational questions that mixed methods researchers ask about their research design that directly affect their sampling design (e.g., What is the purpose of the study?). Then, we move to the tree’s limbs, where we consider the relationship among our research strands/ phases (i.e., identical, parallel, nested and/or multilevel). From each limb, we branch out into

Figure 9.2  A flowchart illustrating the decisions in the tree sampling meta-framework Adapted from Crossover displays for mixed methods sampling designs that are Transparent, Rigorous, Ethical, and Equitable (TREE), by J. A. Corrigan and A. J. Onwuegbuzie, 2022, unpublished manuscript, Concordia University, Montreal, Quebec, Canada, p. 2. Copyright 2022 by J. A. Corrigan and A. J. Onwuegbuzie.

134

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

decisions concerning the combination, emphasis and sequence of our strands/phases (i.e., QUAN + qual + qual; MIXED à QUAL). In the outer branches of the tree, we consider the sampling type (i.e., random/mixed/purposive) and sampling scheme (e.g., stratified random sampling, random purposeful sampling, maximum variation sampling) for each phase/strand. Finally, as we move into the twigs, we consider any other sampling units (e.g., of groups, time, events, items) that have yet to be considered. Depending on our roots, and then considering the trunk of the tree that we use as a foundation, each limb and branch represents a series of choices that we make, which ultimately affect the quantity and quality of the data that we collect, represented by the tree’s leaves. We will demonstrate the tree meta-framework in action as we describe our latest joint research study—namely, University life in an era of disruption of COVID-19: Perception of readiness and attitudes of university students. In this international study (spanning five continents with a site in Montreal, Canada), we used questionnaires and interviews to investigate the effect of COVID-19 lockdowns on teaching, learning and leadership among higher education faculty, students and administrators, respectively. For a further example of the impacts of COVID-19 on higher education contexts, see Chapter 19).

THE ROOTS: EXAMINING OUR PARADIGMATIC ASSUMPTIONS We begin with our paradigmatic assumptions because they influence our entire study design, including our sampling decisions. In Figure 9.2, we provide a flow chart demonstrating the decisions in the TREE sampling process, which begins with an examination of paradigmatic assumptions. Researchers approach studies with certain paradigmatic assumptions (e.g., some form of pragmatism, postpositivism, constructivism), consciously or not, which influence their decisions around, for example, which research questions to ask in the first place, the methods used to answer those questions, what counts as evidence, and what constitutes a valid/representative sampling scheme and size. By paradigm, we are referring to researchers’ worldview, along with the system of beliefs and practices that they associate with it (Kuhn, 1962). For example, for constructivist-based (mixed methods) researchers, the goal of at least the qualitative phase of a mixed methods research study in general and a qualitative-dominant mixed methods research study in particular is not to

produce findings that are generalized from the underlying sample to the population from which the sample was drawn (i.e., external statistical generalizations; Onwuegbuzie et  al., 2009). Therefore, for a constructivist-based (mixed methods) researcher, sampling schemes ought to be purposive rather than random. On the other hand, some researchers advocate for a dialectic stance, which acknowledges that there are multiple ways of seeing, hearing, and knowing the world. Therefore, researchers holding a dialectic stance value multiple and diverse paradigms and their inherent practices (Greene & Hall, 2010). Researchers holding such a stance might select a random sample for one phase of their study while selecting a purposive sample in the next phase, as we did in our study on the impact of COVID-19 on post-secondary students in Montreal.

THE TRUNK: ARTICULATING THE DESIGN OF OUR STUDY What is the Purpose of Your Study and Your Purpose for Mixing? In our tree meta-framework, we now move from considering our roots (i.e., paradigmatic assumptions) to the trunk of the tree, wherein we consider the major research design decisions that support the remainder of the study. They have an important influence in how we select our sampling design (i.e., the branches), the data we collect (i.e., the leaves), and the eventual inferences and conclusions we can make. Most studies begin with a researcher considering the purpose or objective of their research. Johnson and Christensen (2020), for example, describe six commonly cited objectives for research: • Exploration: generating new ideas about phenomena. • Description: describing the characteristics of a phenomenon. • Understanding: understanding the subjective viewpoints of particular people and particular groups. • Explanation: showing how and why a phenomenon operates as it does by studying cause-andeffect relationships. • Prediction: predicting or forecasting a phenomenon using cause-and-effect operations. • Influence: the pursuit of applying research to make certain outcomes occur.

USING A TREE METAPHOR AS A SAMPLING META-FRAMEWORK

In the context of sampling, the objective is important because it helps to determine the emphasis between the qualitative phase(s)/strand(s) and the quantitative phase(s)/strand(s). For instance, if the objective is to explore, then it is likely that a qualitative-dominant mixed methods research study would be conducted that necessitates a small, purposive (i.e., non-random) sample, at least in the qualitative phase(s)/strand(s). Contrastingly, if the objective of the study is to predict, then it is likely that a quantitative-dominant mixed methods research study would be conducted that necessitates a large, random sample, at least in the quantitative phase(s)/ strand(s). As a real example, in our COVID-19 study, our objective is to describe the impact of the COVID-19 pandemic on post-secondary students. Specifically, we want to describe the obstacles faced by students during the pandemic, how they are struggling, and why they are struggling; and we want to understand whether these obstacles are systemic in nature. While we consider the purpose for our study, as mixed methods researchers, we also need to consider our purpose for mixing. According to Collins et  al.’s (2006) typology of rationales for conducting mixed methods research, there exist four major rationales for mixing: • participant enrichment (i.e., recruit participants; engage in activities such as Institutional Review Board debriefings; ensure that each participant selected is appropriate for inclusion); • instrument fidelity (i.e., assess the appropriateness and/or utility of existing instruments; create new instruments); • treatment integrity (i.e., assess fidelity of intervention); and • significance enhancement (i.e., facilitate depth and breadth of data; improve interpretation of findings). In our COVID-19 study, we are using the quantitative survey phase to recruit participants for the qualitative interview phase (i.e., participant enrichment), and we want to deepen our understanding of the quantitative results via thick descriptions (Geertz, 1973) resulting from the qualitative interviews (i.e., significance enhancement).

Whom are you Studying? Now that we have established our purpose for the study (description) and our purpose for mixing (participant enrichment and significance

135

enhancement), we need to describe clearly our participants—in other words, whom we are studying (i.e., sample/population). The COVID-19 study is occurring at a number of sites throughout the world, one of which is in Montreal at Concordia University. In order to select a sample, we typically begin by creating or locating a sampling frame, which is a “formal or informal list of units or cases from which the sample is drawn” (Teddlie & Tashakkori, 2009, p. 180). For example, if we are interested in drawing a sample from students at Concordia University, it would be beneficial to have a list of all of the currently registered students, along with their demographic characteristics and programme status. We would also need to decide if we wanted to conduct a census (the entire population, in this case, all of the students registered at Concordia) or to draw a sample (a subset of the population). Because Concordia University has approximately 50,000 students, conducting a census is likely not feasible due to the time and money this would require. It is also unnecessary because, if we can obtain a large, representative sample—via a statistical power analysis, then we can generalize from the sample to the population (i.e., external statistical generalization). For these reasons, researchers working with large populations typically opt to study a sample of the population instead of the whole (target) population.

What Approaches (i.e., Quantitative vs. Qualitative vs. Mixed) are You Using? How Many Phases/Strands of Each? After deciding what and whom we are studying, we need to consider what research approach is most appropriate for each phase/strand: qualitative, mixed or quantitative. In our study on COVID 19’s impact on post-secondary students, we are using a mixed approach in the first phase (i.e., the questionnaire contains both open-ended [qualitative] and select-response [quantitative] items) and a qualitative approach for the second phase (i.e., semi-structured interviews). This mixed approach of the questionnaire will enable us to collect both a depth and breadth of data, whereas the subsequent in-depth interviews will enable us to enhance the interpretation of our findings by providing thick descriptions. Using a mixed methods research approach for the overall study will enable us to leverage the inherent strengths of both methods while limiting the weaknesses inherent in any one method (Greene, 2007)— what Johnson and Turner (2003) referred to as

136

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

the “fundamental principle of mixed method research” (p. 299).

What Type of Research Design(s) are You Using in Each Phase/Strand? What Optimal Sample Size(s) Is Associated With This Design? The selection of the research design (e.g., experimental, case study, ethnography)—arguably more than any other consideration—is pivotal in determining the optimal sample size. For multiphase and multistrand research studies, the research design has to be decided for each phase or strand, along with the accompanying sample size. Each research design has accepted sample size traditions, based on the norms, values and practices of the researchers who have used these designs and methods (often for decades), which are documented in the literature. That is why it is incumbent that researchers familiarize themselves with the normative practices associated with their particular research designs and methods. In our COVID-19 study, we intend to use a correlational design for the first phase of the study so that we can determine the strength and direction of the linear relationship between variables (e.g., What is the relationship between Internet accessibility and motivation for online learning?). As an example, correlational studies require a minimum of 64 participants for a one-tailed hypothesis or 82 for a two-tailed hypothesis (Onwuegbuzie et al., 2004). Sample size tables—found in some textbooks and journal articles—are helpful tools for determining sample size requirements for common research designs (e.g., Table 4: Minimum sample size recommendations for selected qualitative and quantitative research designs: Corrigan & Onwuegbuzie, 2020, pp. 810–811). Additionally, for the quantitative phase or strand of a study, there exist a number of free online calculators (e.g., G* Power; Faul et al., 2007) that can assist a researcher in performing an a priori power analysis, which involves “estimating the sample size required for a study based on predetermined maximum tolerable Type I and II error rates and the minimum effect size that would be clinically, practically, or theoretically meaningful” (Kyonka, 2019, p. 1) and also a post-hoc power analysis, which involves determining “the proportion of the noncentral distribution that exceeds the critical value used to define statistical significance” (Murphy & Myors, 1998, p. 24). Although sampling considerations are less established in qualitative research designs, there exist a number of published guidelines with

regard to sample sizes across common qualitative designs and methods (e.g., see again Corrigan & Onwuegbuzie, 2020). In the second phase of our COVID-19 study, we are conducting semistructured interviews. But how many interviews (or focus groups, qualitative observations, etc.) are enough? This is an important question that has been explored in the literature (Baker & Edwards, 2012; Beitin, 2014; Guest et  al., 2006) and a common challenge faced by mixed methods researchers during the qualitative phase or strand of their studies. A well-accepted criterion for determining a sample size in a qualitative research study is that of theoretical saturation—the point at which additional data provide no new information such that the emergent theory is adequately developed to fit any future data collected (Glaser & Strauss, 1967). Some even have referred to theoretical saturation as representing the gold standard (Guest et al., 2006) for determining sample sizes in qualitative research. However, critics argue that the use of theoretical saturation to determine the adequacy of a sample size is encumbered by a lack of uniform standards describing how and when saturation is reached (Beitin, 2014). Therefore, in the absence of uniform standards for using saturation to determine sample size, researchers should consider the heterogeneity of the population from which the sample is drawn (Baker & Edwards, 2012; Guest et  al., 2006); the breadth of the research objectives (Baker & Edwards, 2012); the inclusion of multiple perspectives and multiple selves (Beitin, 2014); accuracy; and feasibility (Kemper et al., 2003). The heterogeneity of the population from which the sample is drawn has a direct impact on the sample size needed to reach saturation (Baker & Edwards, 2012; Guest et al., 2006). The more homogeneous the population, the smaller the sample size needed for saturation and vice versa. This was demonstrated in a study by Guest et  al. (2006), who systematically documented the degree of saturation after each set of six interviews for a total of 60 interviews with female sex workers from West Africa (specifically, Ghana and Nigeria). The authors noted that 73 per cent of the total number of codes were generated during the first six interviews, and 92 per cent of the codes after 12 interviews. Somewhat surprisingly, during the second set of 30 interviews with sex workers from Nigeria (the first 30 were from Ghana), no substantive codes emerged such that incoming data “offered few novel insights” (p. 68). The implication here is that the perceptions of female sex workers in these two countries are quite similar. Not only were the participants from this study relatively homogeneous (e.g., same gender, same occupation, similar age, similar geographic location), but also the

USING A TREE METAPHOR AS A SAMPLING META-FRAMEWORK

research objectives were tightly focused (understanding how female sex workers talk about, and perceive, sex). One could imagine that, for a much broader study with a more heterogeneous sample, it would take a greater number of interviews to achieve saturation. Therefore, the breadth of the research objectives should be considered when determining the sample size. Beyond considering the heterogeneity of the population from which the sample is drawn and the scope of the research, researchers also might consider recruiting a sample of participants with multiple perspectives and multiple selves in order to achieve greater representativeness—in other words, using a sample to represent as many perspectives as possible (Beitin, 2014; O’Leary, 2017). For example, in Mishna’s (2004) study of children’s experiences of victimization by bullying, a much more nuanced definition of bullying was achieved by interviewing not only students, but also parents, teachers and administrators. When seeking out multiple perspectives, a researcher might consider variables, such as social role, occupation, gender, race, age and socioeconomic status. Similarly, researchers can also solicit multiple perspectives from the same participants because people simultaneously identify with multiple identities or multiple selves (Holstein & Gubrium, 1995). For example, in our investigation, we will interview participants regarding how COVID-19 has impacted them in their role as a student as it intersects with their roles as a parent (caring for young children during school closures while simultaneously taking classes), child (caring for elderly parents), employee, spouse and/or friend. Asking students about the intersectionality of their multiple identities will undoubtedly provide us with a much fuller picture. Ensuring broad representation—namely, “representation wide enough to ensure that an institution, cultural group or phenomenon can be spoken about confidently” (O’Leary, 2017, p. 144)—is an important technique that helps enhance the rigour and equitableness of the study. Next, when determining the sample size, researchers should consider the need for accuracy, defined as the dependability and truthfulness of our research findings (Yarbrough et al., 2010). In the aforementioned example of sex workers, interviewing six participants might be acceptable if the researchers were conducting some preliminary exploratory research with the goal of better understanding the experiences of sex workers. But if their goal was to use these data to create a programme to prevent increasing rates of HIV infection among sex workers (and presumably this programme would require a great deal of resources), then the stakes are higher and the

137

need for accuracy increased. The greater the need for accuracy, the greater the sample size needed. Yet, sometimes the need for accuracy is chastened with the need for feasibility or, in other words, the need to be both effective and efficient. For example, although a researcher might ideally want to interview 60 participants, there might not be the time or the resources to justify such a sample size. Therefore, determining an appropriate sample size represents an ongoing challenge to balance feasibility and accuracy.

THE LIMBS: IDENTIFYING THE SAMPLING DESIGN FOR EACH PHASE/STRAND What is the Relationship Among the Phases/Strands? The trunk of the tree branches into limbs that each represent a phase/strand of the study. It is up to the researcher to determine what the relationship will be among the phases/strands. Onwuegbuzie and Collins (2007) proposed the terms “identical”, “parallel”, “nested” and “multilevel” to refer to the relationship among/between the samples: • Identical: The same sample is used across all phases/strands (e.g., all participants both complete a questionnaire and participate in an interview). • Parallel: The samples used between two phases/ strands are different, but are drawn from the same underlying population (e.g., an intervention study is piloted in one school board and, in the next phase, is implemented in a school board with similar characteristics). • Nested: The sample selected for one phase/strand is a subset of the sample used for another phase/ strand (e.g., focus group participants are selected from survey respondents who meet the inclusion criteria). • Multilevel: Two or more sets of samples are selected from different levels of the same investigation (e.g., students, teachers, principals, school boards). Inherent in mixed methods researchers’ decisions around the relationship between/among samples that represent each phase of the study are a number of fundamental choices. For instance, when selecting parallel samples, researchers are seeking two or more groups with similar characteristics—otherwise, researchers will end up comparing proverbial apples to oranges. As an

138

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

example, if the quantitative sample(s) comprises only men or a predominance of men, and the qualitative sample(s) comprises only women or a predominance of women (or vice versa), then, in the case wherein there is a true gender context to the phenomenon being studied, meta-inferences stemming from combining the inferences from the quantitative sample(s) and inferences from the qualitative sample(s) would be untrustworthy, especially if a/the goal of the meaning-making process was to triangulate the findings from the two (sets of) samples. We argue that mixed methods researchers almost always lack transparency in describing the relationship between/among samples and, hence, give the impression that their sampling designs lack (sufficient) rigour. This is not surprising considering how little guidance exists in the literature in this area. However, the time has come for mixed methods researchers to become more rigorous and transparent about their sampling designs when sampling among/between phases/ strands. Therefore, we call attention to this neglected area by bringing attention to the work of Onwuegbuzie and Corrigan (2021), who proposed matching guidelines for selecting samples for each of the four types of relationships among/ between phases/strands—namely, identical samples, parallel samples, nested samples and multilevel samples—such that the different samples are linked in an attempt to ensure that they are optimally comparable.

What Level of Mixing will You Use? What Combination, Emphasis, and Sequence of Phases/Strands will You Use? Now that the researcher has established the relationship between/among the samples, the next step is to determine the level of mixing, and then the combination, emphasis and sequence of phases/ strands. Research studies exist on a continuum, from not mixed (i.e., monomethod method studies that use an exclusively quantitative or qualitative approach) to fully mixed studies, with partially mixed studies occupying the space along that continuum (Leech & Onwuegbuzie, 2009). There is a misconception among researchers that mixed methods research studies—as their name implies— are mixed only at the level of methods. However, for the past two decades, mixed methods researchers have been advocating for mixing at many stages of the enquiry process, such as mixing paradigms; mixing research purposes and questions; and mixing at the level of data collection and

analysis (Greene, 2007), prompting some to use the term mixed research instead of mixed methods research (Johnson et  al., 2007). Moreover, in recent years, members of the mixed methods research community have changed from discussing the level of mixing to discussing the level of integration (Onwuegbuzie, 2017). As observed by Onwuegbuzie and Hitchcock (2019): each mixed research study can be placed somewhere on an integration continuum, depending on the degree to which quantitative and qualitative assumptions, approaches, frameworks, methods, techniques, concepts, language, and the like— that often are associated with either the qualitative or quantitative research tradition—are integrated and interact with each other during the study. (pp. 8–9)

Researchers who conceptualize and plan studies that represent what Onwuegbuzie and Hitchcock (2019) refer to as “the full integration of qualitative and quantitative components at the data collection, data analysis, and data interpretation stages of the mixed methods research process” (p. 10) clearly would benefit from a tree approach to sampling in order to integrate inferences from both the quantitative and qualitative phases/ strands in an attempt to generate meta-inferences (Corrigan & Onwuegbuzie, 2020). This is because the full(er) integration of qualitative and quantitative components yields (more) integrated sampling designs that are potentially more complex, thereby necessitating transparency (e.g., delineating the integrated sampling design clearly, ideally via visual representation [e.g., Figure 9.2]), rigour (e.g., ensuring that optimum matching of integrated samples takes place; Onwuegbuzie & Corrigan, 2021), equitableness (e.g., ensuring fairness in the integrated sampling process) and ethicality (e.g., adhering to critical dialectical pluralism-based principles of maximizing non-maleficence [i.e., not causing harm to participants] and beneficence [i.e., working for the benefit of participants and other stakeholders] throughout the integrated sampling process). The need for using the tree approach is driven by the need to obtain what we have defined as representation—namely, the appropriate balance of quantitative power and qualitative saturation (Corrigan & Onwuegbuzie, 2020). Next, we consider the combination, emphasis, and sequence of the phases/strands. With regard to sampling combination, Onwuegbuzie and Collins (2007) have devised the following schema: concurrent–identical, concurrent–parallel, concurrent–nested, concurrent–multilevel, sequential– identical, sequential–parallel, sequential–nested

USING A TREE METAPHOR AS A SAMPLING META-FRAMEWORK

and sequential–multilevel. We have already described the second portion of these hyphenated terms (i.e., identical, parallel, nested, multilevel) and now we will describe the first portion. The terms “concurrent” and “sequential” are often misconstrued as referring solely to a time dimension of the study (i.e., Phase X happened before Phase Y; therefore, this must be a sequential design). A more accurate way to conceptualize a sequential sampling design is to consider whether sampling from Phase Y depends on the sampling from Phase X. For example, in our COVID-19 study, we are using our survey data to purposefully select the participants for our interviews. Thus, the interview sample is dependent on the survey sample, which yields a sequential design. Conversely, in a concurrent sampling design, the samples can be chosen at the same time or at different times because they do not depend on one another. These sampling combinations also connect back to our purpose for mixing methods. For example, if our purpose for mixing methods is triangulation, then a concurrent (i.e., independent) sampling design is required to use findings emanating from a sample representing a subsequent phase(s)/strand(s) to validate/legitimate findings stemming from a sample representing an earlier phase(s)/strand(s). The emphasis of the research design refers to whether the phases/strands have approximately equal priority (i.e., equal status) with respect to answering the research question(s), or whether one has significantly higher priority (i.e., dominant status; Leech & Onwuegbuzie, 2009). Consider that the research purpose(s) can generally point towards which phase/strand of the study has priority. For example, in our COVID19 study, if our main purpose is to predict which students are most at risk for attrition based on a number of variables (e.g., Internet connection, motivation for online learning), then our emphasis would be on the quantitative (i.e., survey) phase, necessitating a large, random sample. Combining the mixing dimension (partially vs. fully mixed), time dimension (concurrent vs. sequential), and emphasis dimension (equal vs. dominant status), Leech and Onwuegbuzie (2009) have devised a three-dimensional typology for research designs (e.g., fully mixed sequential equal status design), which Guest (2013) describes as “laudable”, making “an important contribution” and “reduc[ing] much of the confusion surrounding the vague boundaries between designs that is typical of other typologies” (p. 145). In order for researchers to add greater transparency to the reporting of their research, we recommend that they specifically state these dimensions of their research design and align them with their sampling design.

139

What Sampling Technique and Sampling Scheme Will You Use for Each Phase/Strand? There are three major sampling techniques: random (i.e., “a sample drawn by a procedure in which every member of the population has an equal [and independent] chance of being selected” [Johnson & Christensen, 2020, p. 242]), convenience (i.e., “including people who are available, volunteer, or can be easily recruited in the sample” [Johnson & Christensen, 2020, p. 253]), and purposive sampling (“the researcher specifies the characteristics of the population of interest and locates individuals with those characteristics” [Johnson & Christensen, 2020, p. 254]). To this, we would add a fourth technique: mixed sampling, which simultaneously involves both random sampling and purposive sampling, such as multi-stage purposeful random sampling. As described by Onwuegbuzie and Collins (2007), multi-stage purposeful random sampling involves selecting settings, groups and/or individuals that represent a sample in two or more stages: the first stage involving random selection and the following stages involving purposive selection of participants. Deciding whether to select a random, convenience, purposive or mixed sampling technique depends largely on the generalization goal. For example, in our COVID-19 study, if we wanted to generalize from the sample of survey participants to the university population (i.e., external statistical generalization; Onwuegbuzie et  al., 2009), then, optimally, a large and random sampling strategy would be necessitated. For some other types of generalizations—internal statistical (i.e., to generalize conclusions to the sample from the [ideally representative] sub-sample; Onwuegbuzie et al., 2009), analytic (i.e., to generalize case study results to some broader theory; Onwuegbuzie et  al., 2009) and case-to-case transfer (to generalize from one case to another [similar] case; Onwuegbuzie et al., 2009)—however, a purposive sampling strategy would be more appropriate. In a previous article, we have provided a table considering generalization goals alongside sampling strategies and sample sizes/number of sampling units (Table 1: Generalization goals and accompanying recommendations for sample size/number and assumptions about the sample—Corrigan & Onwuegbuzie, 2020). Additionally, within each of the four types of sampling techniques are a number of sampling schemes (e.g., stratified random sampling, maximum variation) that describe in greater detail the sampling strategy (see Table 3: Major sampling schemes in mixed methods research—Corrigan & Onwuegbuzie, 2020).

140

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

What are the other sampling units? A final step in using the tree sampling metaframework is to consider all other sampling units. Although individual people are the most commonly considered element sampled from a population, researchers also sample groups (e.g., families, schools, classrooms), research units (e.g., number of interviews, number of interview questions, number of survey items), time (e.g., number of minutes for an observation, number of minutes for an interview), events (e.g., number of visits to a site for a prolonged observation; number of traffic stops; number of social media shares, number of experiences, incidents, activities), items (e.g., number of Tweets, number of articles, number of words), and the like. For each of these units, once again, the researchers should consider selecting an appropriate sample size and sampling scheme.

rigorous, equitable and ethical), we contend that our sampling meta-framework will help to move further towards what Onwuegbuzie et al. (in press) refer to as procedural justice and process justice that promote both universalistic theoretical knowledge and local practical knowledge—for example, by creating spaces for the voices of the underrepresented, underserved, disenfranchised and marginalized persons or groups to be appropriately represented via a transparent, rigorous, equitable and ethical matching process. We recognize that our critical dialetical pluralism-based tree sampling meta-framework adds a layer of complexity and complication to the mixed methods research sampling process. However, we believe that this layer is offset by the fact that this meta-framework provides a pathway for mixed methods researchers to improve their current sampling practices. We hope that this chapter represents an important step towards providing a tree sampling approach that has the potential to grow as other researchers and methodologists build on the foundation that we have provided.

CONCLUSIONS AND FUTURE STEPS Trees are perennial plants that live for many years. And, indubitably, in an era of post-truth, wherein the authority of (professional) researchers in general and their knowledge production in particular are being challenged and delegitimized worldwide (Wolgemuth et  al., 2018), an important goal of mixed methods researchers is to maximize the life cycle of the mixed methods research field—to ensure that mixed methods research is perennial. And one important step towards achieving this is by conducting research that has the attributes of being transparent, rigorous, equitable and ethical. Unfortunately, to date, in general, the sampling designs of mixed methods research studies have lacked one or more of these attributes, thereby (potentially) adversely affecting the integrity of the sampling design and, in turn, interpretive consistency, which represents the justifiableness of the type of generalization made, given the sampling design. With this in mind, in this chapter, we have provided a critical dialetical pluralism-based, metaframework for formalizing the mixed methods research sampling process. In providing our metaframework, we have planted a tree—specifically, a tree sampling meta-framework. In introducing and promoting our tree sampling meta-framework, we are providing an alternative model for conceptualizing, planning and implementing sampling designs that are transparent, rigorous, equitable and ethical. Further, following the four-element tenets of the TREE approach (i.e., transparent,

WHAT TO READ NEXT Onwuegbuzie, A. J., & Collins, K. M. T. (2017). The role of sampling in mixed methods research: Enhancing inference quality. Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(Supplement 2), 133–156. https://doi.org/10.1007/s11577-0170455-0

In this article, the authors emphasized the importance of sampling in all mixed methods research studies, noting that the quality of the mixed methods sampling design determines the quality of inferences that emerges, which, in turn, determines the quality of the meaning-making process. According to these authors, the quality of inferences is dependent on whether interpretive consistency occurs. This article is particularly useful because it complements our tree sampling meta-framework. Corrigan, J. A., & Onwuegbuzie, A. J. (2020). Toward a meta-framework for conducting mixed methods representation analyses to optimize meta-inferences. Qualitative Report, 25(3), 785–812. https:// doi.org/10.46743/2160-3715/2020.3579

In this article, the authors present a meta-framework for conducting what they term “mixed methods representation analyses” (MMRA), which they define as the appropriate selection of sampling design (i.e., the sampling frame [random] or

USING A TREE METAPHOR AS A SAMPLING META-FRAMEWORK

sampling boundary [purposive]; sampling combination, comprising the mixing dimension [partial/ fully], time dimension [concurrent/sequential], emphasis dimension [dominant/equal status], and relationship among/between samples [identical/ parallel/nested/multilevel]; sample size; and number of sampling units [e.g., of people, cases, words, texts, observations, events, incidents, activities, experiences, or any other object of study]) in order to obtain representation and interpretive consistency that enhances the rigour of mixed methods research studies. This article is particularly useful because it also complements our tree sampling meta-framework. Onwuegbuzie, A. J., & Corrigan, J. A. (2021). Matching techniques in mixed methods research-based studies: The role of critical dialectical pluralism. International Journal of Multiple Research Approaches, 13(2), 116–136. https://doi.org/ 10.29034/ijmra.v13n2editorial2

In this article, the authors outline how mixed methods researchers can become more rigorous and transparent about their sampling designs when sampling among/between phases/strands. In particular, these authors discuss rigorous matching techniques for each of the following four types of relationships among/between phases/strands: identical samples, parallel samples, nested samples and multilevel samples. This article is particularly useful because it builds on our tree sampling meta-framework.

REFERENCES Baker, S. E., & Edwards, R. (2012). How many qualitative interviews is enough: Expert voices and early career reflections on sampling and cases in qualitative research. In National Centre for Research Methods Review Paper. https://eprints.ncrm.ac.uk/ id/eprint/2273/4/how_many_interviews.pdf Bazeley, P., & Kemp, L. (2012). Mosaics, triangles, and DNA: Metaphors for integrated analysis in mixed methods research. Journal of Mixed Methods Research, 6(1), 55–72. https://doi.org/ 10.1177/1558689811419514 Beitin, B. K. (2014). Interview and sampling: How many and whom. In The SAGE Handbook of interview research: The complexity of the craft (pp. 243–254). Sage. Canada Research Coordinating Committee. (2021). Best practices in equity, diversity and inclusion in research. A guide for applicants to New Frontiers in Research Funds competitions. Government of Canada. www.sshrc-crsh.gc.ca/funding-financement/ nfrf-fnfr/edi-eng.aspx

141

Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L. (2006). A model incorporating the rationale and purpose for conducting mixed-methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4(1), 67–100. Corrigan, J. A., & Onwuegbuzie, A. J. (2020). Toward a meta-framework for conducting mixed methods representation analyses to optimize meta-inferences. Qualitative Report, 25(3), 785–812. https:// doi.org/10.46743/2160-3715/2020.3579 Corrigan, J. A., & Onwuegbuzie, A. J. (in press). Using a transparent, rigorous, ethical, and equitable (TREE) sampling meta-framework to enhance mixed methods representation. International Journal of Multiple Research Approaches. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146 Geertz, C. (1973). Thick description toward an interpretive theory of culture. In C. Geertz (Ed.), The interpretation of cultures (pp. 3–30). Basic Books. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine. Greene, J. C. (2006). Toward a methodology of mixed methods social inquiry. Research in the Schools, 13(1), 93–98. Greene, J. C. (2007). Mixed methods in social inquiry. Jossey-Bass. Greene, J. C., & Hall, J. N. (2010). Dialectics and pragmatism: Being of consequence. In A. Tashakorri & C. Teddlie (Eds.), SAGE handbook of mixed methods in social and behavioral research (2nd., pp. 119–144). Sage. Guest, G. (2013). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7(2), 141–151. https:// doi.org/10.1177/1558689812461179 Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough?: An experiment with data saturation and variability. Field Methods, 18, 59–82. https://doi.org/10.1177/1525822X05279903 Hense, A. (2017). Sequential mixed methods sampling: How quantitative secondary data can support qualitative sampling plans and theoretical sampling [Sequentielles mixed-methods-sampling: Wie quantitative sekundärdaten qualitative stichprobenpläne und theoretisches sampling unterstützen können]. Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(Supplement 2), 237–259. https://doi.org/10.1007/s11577017-0459-9 Holstein, J. A., & Gubrium, J. F. (1995). The active interview. Sage. Israel, M., & Hay, I. (2006). Why care about ethics?. In M. Israel & I. Hay (Eds.), Research ethics for social scientists (pp. 2–11). Sage.

142

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Johnson, R. B., & Christensen, L. (2020). Educational research: Quantitative, qualitative, and mixed approaches (7th ed.). Sage. Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1525/sp.1960. 8.2.03a00030 Johnson, R. B., & Turner, L.A. (2003). Data collection strategies in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 297–319). Sage. Kemper, E. A., Stringfield, S., & Teddlie, C. (2003). Mixed methods sampling strategies in social science research. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social and behavioral research (pp. 273–296). Sage. Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press. Kyonka, E. G. E. (2019). Tutorial: Small-N power analysis. Perspectives on Behavioral Science, 42(1), 133–152. https://doi.org/10.1007/s40614-0180167-4 Leech, N. L., & Onwuegbuzie, A. J. (2009). A typology of mixed methods research designs. Quality and Quantity, 43(2), 265–275. https://doi.org/ 10.1007/s11135-007-9105-3 Lincoln, Y., & Guba, E. G. (1985). Naturalistic inquiry. Sage. Mishna, F. (2004). A qualitative study of bullying from multiple perspectives. Children & Schools, 26(4), 234–247. https://doi.org/10.1093/cs/26.4.234 Murphy, K. R., & Myors, B. (1998). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests. Lawrence Erlbaum Associates. O’Leary, Z. (2017). The essential guide to doing your research project (3rd ed.). Sage. Onwuegbuzie, A. J. (2017, March). Mixed methods is dead! Long live mixed methods! Invited keynote address presented at the Mixed Methods International Research Association Caribbean Conference at Montego Bay, Jamaica. Onwuegbuzie, A. J., Abrams, S. S., & Forzani, E. (in press). The many SIDES of critical dialectical pluralism: A meta-philosophy—comprising a research philosophy, educational philosophy, and life philosophy— for addressing social justice, inclusion, diversity, and equity, and social responsibility. International Journal of Multiple Research Approaches. Onwuegbuzie, A. J., & Collins, K. M. T. (2007). A typology of mixed methods sampling designs in social science research. The Qualitative Report, 12(2), 281–316. http://files.eric.ed.gov/fulltext/ EJ800183.pdf

Onwuegbuzie, A. J., & Collins, K. M. T. (2017). The role of sampling in mixed methods research: Enhancing inference quality. Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(Supplement 2), 133–156. https://doi.org/10.1007/s11577-0170455-0 Onwuegbuzie, A. J., & Corrigan, J. A. (2021). Matching techniques in mixed methods research-based studies: The role of critical dialectical pluralism. International Journal of Multiple Research Approaches, 13(2), 116–136. https://doi. org/10.29034/ijmra.v13n2editorial2 Onwuegbuzie, A. J., & Frels, R. K. (2013). Introduction: Towards a new research philosophy for addressing social justice issues: Critical dialectical pluralism 1.0. International Journal of Multiple Research Approaches, 7(1), 9–26. https://doi. org/10.5172/mra.2013.7.1.9 Onwuegbuzie, A. J., & Hitchcock, J. H. (2019). Toward a fully integrated approach to mixed methods research via the 1 + 1 = 1 integration approach: Mixed Research 2.0. International Journal of Multiple Research Approaches, 11(1), 7–28. https://doi.org/10.29034/ijmra.v11n1editorial1 Onwuegbuzie, A. J., Jiao, Q. G., & Bostick, S. L. (2004). Library anxiety: Theory, research, and applications (Research Methods in Library and Information Studies, No. 1). Scarecrow Press. Onwuegbuzie, A. J., Slate, J. R., Leech, N. L., & Collins, K. M. T. (2009). Mixed data analysis: Advanced integration techniques. International Journal of Multiple Research Approaches, 3(1), 13–33. https://doi.org/10.5172/mra.455.3.1.13 Peshkin, A. (1988). In search of subjectivity—one’s own. Educational Researcher, 17(7), 17–21. https://doi.org/10.3102/0013189X017007017 Randall, J., Poe, M., & Slomp, D. (2021). Ain’t oughta be in the dictionary: getting to justice by dismantling anti-black literacy assessment practices. Journal of Adolescent & Adult Literacy, 64(5), 594–599. https://doi.org/10.1002/jaal.1142 Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioural sciences. Sage. Wolgemuth, J. R., Koro-Ljungberg, M., Marn, T. M., Onwuegbuzie, A. J., & Dougherty, S. M. (2018). Start here, or here, no here: Introductions to rethinking education policy and methodology in a post-truth era [Special issue]. Education Policy Analysis Archives, 26(145), 1–8. https://doi. org/10.14507/epaa.26.4357 Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The program evaluation standards: A guide for evaluators and evaluation users. Sage.

10 Data Integration as a Form of Integrated Mixed Analysis in Mixed Methods Research Designs S u s a n n e Vo g l

WHAT IS INTEGRATED MIXED ANALYSIS? Integration in Mixed Methods Research Within the mixed methods research (MMR) community, there is growing sensitivity towards practical and conceptual challenges in meeting the expectations of MMR to gain synergies by using both qualitative and quantitative methods and/or data. Many researchers consider integration to be the hallmark (Moseholm & Fetters, 2017) or “the heart and soul of mixed methods research” (Guetterman et al., 2020, p. 430). Despite the great potential of integration, it often remains underdeveloped and undertheorized, to the extent that many see addressing the “integration challenge” (Fetters & Freshwater, 2015) as the key hurdle for the future of MMR. What does integration mean? Integration is central to producing a sum that exceeds the individual quantitative and qualitative parts (Bryman, 2007), with different elements or strategies becoming interdependent in reaching a common research goal (Bazeley, 2010). In mathematical terms, the outcome of integration can be depicted as 1 + 1 = 3 (Fetters & Freshwater, 2015). However, some scholars suggest 1 + 1 = 1 as a more appropriate mathematical formula to represent integration because the outcome of full integration cannot be

distinguished into separate parts; instead, it becomes a (new) whole (e.g., Onwuegbuzie & Hitchcock, 2019). With regard to practices of integration, different designs allow for different integration strategies and integration can occur at various stages of the research process. For a discussion of building the logic for an integrated methodology, see also Chapter 12 (this volume). In this chapter, data integration as a form of analysis in MMR designs is at focus. Thus, I will first broadly describe the function of data analysis in MMR to then clarify terminologies and specifics of mixed analysis. Bringing these thoughts on integration and mixed analysis together, integrated mixed analysis and data integration are introduced. With an empirical example, I illustrate the contributions and limitations of three approaches to data integration: linkage, transformation and consolidation. In doing so, I problematize the qualitative–quantitative dichotomy and propose the need for more creative and flexible—yet reflective—thinking around integration.

Data Analysis in MMR: Terminology and Practice Data analysis in MMR can take various forms. It usually consists of multiple steps that can be

144

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

performed sequentially, in parallel or in one or more phases, or in an interactive process with different objectives and purposes. It can be case oriented, variable oriented, and/or process or experience oriented (Onwuegbuzie & Combs, 2010). Usually, data from each strand are analyzed with “appropriate” methods (Creswell & Plano Clark, 2018). Then, in the simplest form of integration, illustrative quotes from qualitative interviews complement or supplement results from statistical analyses. Although it can assist in communicating statistical results, “this type of integration strategy is quite limited” (Bazeley, 2012, p. 817) and does not include the integration of data. However, the analysis phase seems to be particularly appealing for integration, but it is probably also the most challenging phase for integration (Onwuegbuzie & Combs, 2010; Yin, 2006). The challenges lie in developing a form of joint analysis that allows diverse data types (Moran-Ellis et al., 2006). Generally, the purpose of data analysis is to reduce and organize raw data/material to enable detecting patterns of relations, connections, trends, or differences (Greene, 2007), and analysis goes hand-in-hand with interpretation. The distinction between data analysis and data interpretation as separate phases in the research process is often blurred. Analysis is rarely free of interpretation and ultimately all data are analyzed “qualitatively”: “in so far as the act of analysis is an interpretation, and therefore of necessity a selective rendering, of the ‘sense’ of the available data” (Bazeley, 2018b, p. 57; see also Akremi et  al., 2018; Baur & Knoblauch, 2018; Bazeley, 2018b, p. 57; N. Fielding & J. L. Fielding, 1986; Maxwell, 2010; Sandelowski et al., 2009). In MMR, researchers additionally have to determine the relation and relative weight of data strands, the level of interaction between strands, the temporal relation between them, and practical approaches to integration (Creswell & Plano Clark, 2018). These decisions are primarily determined by the research question and the purpose of the MMR design (Brannen & O’Connell, 2015; Onwuegbuzie & Hitchcock, 2015). Researchers must have a clear vision of what “dynamic mixes they suggest or permit” (Sandelowski, 2000, p. 254) and to what end. The aim of data analysis in MMR can be external or internal statistical generalizations, analytical generalizations, case-to-case transfer, and/or naturalistic generalization (Onwuegbuzie & Combs, 2010). The purposes—analogous to design and data collection in MMR—can be triangulation/corroboration/convergent validation; complementarity/analytic density; development; initiation; and elaboration/expansion (Caracelli &

Greene, 1993; Greene et al., 1989; Onwuegbuzie & Hitchcock, 2015; Schoonenboom & Johnson, 2017). Integrative strategies are most appropriate for initiation, expansion, and development purposes. For triangulation, in which the analysis of the same phenomenon from different perspectives is undertaken to increase validity, data analysis of different components should be independent (Onwuegbuzie & Combs, 2010). In the literature, a distinction exists between mixed analysis and mixed methods analysis. Mixed methods analysis of data covers qualitative and quantitative analytic strategies under the same framework or similar frameworks, and entails the analysis of qualitative and quantitative data (Onwuegbuzie & Hitchcock, 2015). As a specific variant, mixed analyses are based on using data from one data collection in different analytic strands. This implies that during the data collection either different types of data have been collected or data have been (partially) transformed. However, despite the source of the data being one or multiple data collections, mixed analysis and mixed methods analysis have varying levels of interaction or integration of data types and generally share comparable challenges. As a matter of fact, for the challenge of data integration as in producing a sum that exceeds the individual qualitative and quantitative parts, it does not matter whether data stem from a single data-collection process or multiple collections, or from concurrent or sequential designs. What does make a practical difference is whether data can be linked on a case or unit of analysis level, or on a concept level—in other words, whether different data are available for the same case and can thus be linked “directly”, or whether it stems from independent samples or refers to different aggregate levels— e.g. individual members of a focus group discussion or the focus group as such, and can thus only be linked on a conceptual level. Thus, I do not differentiate between mixed and mixed methods analyses in this chapter.

Integrated Mixed Analysis In practice, qualitative and quantitative strands are most frequently analyzed separately, and the respective findings are then integrated and interpreted together as meta-inferences (Bazeley, 2016; Caracelli & Greene, 1993). Thus, mixed analysis does not need to be and often is not integrative. Instead, integration takes place during the joint interpretation of findings from different strands. However, integrated mixed analysis goes one step further and refers to mixed analysis with

DATA INTEGRATION AS A FORM OF INTEGRATED MIXED ANALYSIS

an integration component. It asks about the relatedness, or degree of “mutual illumination” (Bryman, 2007), and it occurs to the extent that different data elements become interdependent in reaching a theoretical or research goal (Bazeley, 2010). Integrative mixed analysis is often an iterative process, with analysis and interpretation going hand-in-hand. Generally, starting points for integrated mixed analyses can be the combination of various sources of raw data without prior processing, of pre-processed data that have been coded or indexed, or during the analytic process when different approaches interact (Bazeley, 2018b). Integrated mixed analysis, in contrast to mono-method analysis, not only involves the analysis of a qualitative and a quantitative strand, but it also entails analysis of the interaction between the two strands and potentially a new, integrated/consolidated data set based on merging different data types. Integrated analysis offers additional insights because different analytic strategies can broaden the findings and offer a more nuanced understanding, and take full advantage of multiple research approaches. This approach has notable consequences for the workload and the skills required. Integrated analysis requires more than double the analytic effort, but the interaction or joint analysis of strands adds an entirely new dimension to the analysis. To sum up: mixed analysis entails qualitative and quantitative analysis of qualitative and quantitative data from one or more data collections under the same or a similar framework. Integrated mixed analysis goes one step further because data elements become interdependent: qualitative, and quantitative data are not only analyzed separately but also in their interaction. Data integration is one strategy of integrated analysis, which I will discuss in more detail in the following section. First, I conceptualize the role of data and data integration strategies—specifically, transformation and consolidation. With an empirical example from my own work and the methodological reflection of research practice, I then illustrate the practicalities of an integrated mixed analysis and reflect on the effect of conceptualizing data as qualitative and quantitative in MMR.

DATA IN MIXED ANALYSIS AND DATA INTEGRATION Data take a crucial position within the empirical research process and deserve further attention when it comes to mixed analysis. By definition, mixed analysis deals with at least two different

145

types of data. In integrated analysis, these data types are not only analyzed separately, but also jointly through comparison, linkage, transformation or consolidation. Data integration implies that a joint product is constructed from different sources of data by using various analytical strategies (Li et  al., 2000). Thus, data integration is a specific form of integrated mixed analysis. The challenge of integrated analysis and data integration lies in developing a joint analysis for diverse data sets (Moran-Ellis et  al., 2006). As stated by Lieber and Weisner (2010, p. 567): “When it comes to integrating data for both analysis and interpretation, a key question is how to get the data to ‘speak the same language.’” Through data integration, the results have the potential to produce insights that would not be possible through merely combining data, let alone achieved with mono-method research. Strategies for data integration have included intensive case analysis, typology development, joint display, data transformation, extreme case analysis, following a thread, and creating blended variables for further analysis (Bazeley, 2009; Caracelli & Greene, 1993; Greene, 2007; Johnson et  al., 2017; Moran-Ellis et  al., 2006; Teddlie & Tashakkori, 2009). In the following, data transformation, linkage, and consolidation will be described in more detail.

Data Transformation Data transformation implies that one data type is transformed into another type for the purpose of being analyzed together with other information. Data integration (usually) requires data transformation to bring other types of original data into a compatible format. Qualitizing and quantitizing are core strategies for data transformation and allow for joint analyses of two (or more) research strands and thus data integration. Qualitizing can enable gaining additional insights from quantitative data or testing the interpretation. For example, a typology of respondents can be based on individual values on scales (Bazeley, 2012; Sandelowski, 2000), and this typology or case profiles can be presented as a narrative description based on the most frequent attributes. This is a common strategy for describing samples; however, usually, it is not called “qualitizing” (Sandelowski, 2000). Panel studies have additional potential in that respect because event history models sequences and patterns that can help categorize individuals or groups (Bazeley, 2012). A complex form of qualitizing can be found in Bourdieu’s (1986) work; based on quantitative indicators, he maps the social space (Kuckartz, 2017).

146

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Quantitizing is more frequently employed in MMR. It involves transforming qualitative data to numerical data (Sandelowski, 2000; Sandelowski et al., 2009) to identify patterns or peculiarities in the data by data reduction. Quantitizing leads to data reduction, which can help make large qualitative data sets more manageable, facilitate comparisons with other data sources, and allow statistical analysis, which adds power and sensitivity to individual judgement (Bazeley, 2010; Sandelowski et al., 2009). Thus, patterns and peculiarities that would be hidden or could not be communicated otherwise—particularly in large data sets—can be detected. As with any data-reduction process (e.g., categorization), quantitizing data involves a tradeoff between retaining detail and managing a complex data set (Bazeley, 2018a). Quantifications cannot replace qualitative analysis, but they can be complementary or instigate new hypotheses about relationships and thus advance theoretical thinking. Using counts of themes or codes to enhance the presentation of results or gauge the strength of a phenomenon has a long tradition. For example, thematic analysis often counts codes assigned to text. Counting is a form of description and frequencies of codes/ themes could indicate the (relative) importance of a topic. Codings can be transformed to dichotomous or continuous variables and used in exploratory, explanatory, comparative or confirmatory analysis (Bazeley, 2010; Sandelowski et al., 2009), and the results can complement or corroborate qualitative findings. For example, in an exploratory factor analysis, we can detect thematic patterns in qualitative data (Onwuegbuzie, 2003). In addition, qualitative interview data can be transformed into numerical ratings that are integrated into a quantitative data set to be analyzed together. In this way, one can analyze the relations between personality traits and the evaluation of an issue. Both examples illustrate data transformation. Nevertheless, questions about the extent to which statistical models can be combined with interpretive approaches are challenging because qualitative studies are mostly based on a small number of cases and nonrandom sampling (Fakis et al., 2014; Niglas et al., 2008), but even standardized surveys often do not fulfill this criterion (Bazeley, 2010). However, nonparametric tests, multidimensional scaling, cluster and correspondence analysis can be used because they have only a few or no requirements regarding the data. These procedures do not test hypotheses but instead support exploratory analysis. Statistical techniques producing visual displays of relationships can assist interpretive analysis, as can be seen in correspondence analysis and the mapping of social distinction (e.g., see Blasius & Friedrichs, 2008; Kutscher & Howard, 2021).

Quantitizing also has some challenges; specifically, numbers have a cultural, rhetorical appeal that can suggest greater generalizability for the conclusions than is justified. They “can be used rhetorically, to make a report appear more precise, rigorous, and scientific, without playing any real role in the logic of the study and thus misrepresenting the actual basis for the conclusions” (Maxwell, 2010, pp. 479–480). Furthermore, there is a challenge of reducing evidence to the amount of evidence available. In addition, compromises, judgements, and foundational assumptions in quantitizing qualitative data are typically glossed over: Among these assumptions are that qualitative and quantitative data constitute two kinds of data, that quantitizing constitutes a unidirectional process essentially different from qualitizing, and that counting is an unambiguous process. Among the judgments are deciding what and how to count. Among the compromises are balancing numerical precision with narrative complexity. (Sandelowski et al., 2009, p. 208)

To counter these problems, researchers “must have a clear sense of why they want to quantitize at all. They must consider what the added value is of transforming qualitative into quantitative data” (Sandelowski et  al., 2009, p. 219). The validity of counts depends on the rigidity and consistency of the coding process, as with any numerical data representing the measurement of a phenomenon. Quantitizing should meet some criterion such as clear definition of the codes and consistency in coding across the entire dataset—for example, intercoder reliability can serve to assess consistency in coding. Furthermore, Bazeley (2018b) suggests to illustrate meaning attached to each code with examples of how it was applied, to review data sources coded early in the project to ensure consistency, especially if the coding system was evolving during the course of a project, and to check the consistency of codings by reviewing the text assigned to each code. Assuming these criteria are met, quantitizing can facilitate an effective communication and analytical integrity (Bazeley, 2018b; Maxwell, 2010), and it can assist in showing the complexity and multidimensionality of qualitative data (Sandelowski et al., 2009).

Data Linkage Linking data during analysis is another data integration strategy. It refers to the “combination of data through association, comparison, or

DATA INTEGRATION AS A FORM OF INTEGRATED MIXED ANALYSIS

relational analyses” (Bazeley, 2018b, p. 137), but the original data can still be separated. The advantages of data linkage are comparisons and relational purposes. This means that data is matched for individual cases to compare or combine information from different sources. Patterns and differences, as well as diverging individual cases, can be detected more reliably and the relevance of grouping variables can be revealed. Different sources can work together to constitute a complementary account for rich descriptions of social phenomena, or to reveal differences and patterns across subgroups that might otherwise be obscured (Bazeley, 2018b).

147

design from my own research. This example illustrates integrative strategies of data linkage, transformation, and consolidation to create new qualitative data, as well as blurred boundaries between qualitative and quantitative data and terminology challenges.

EMPIRICAL EXAMPLE FOR INTEGRATION THROUGH TRANSFORMATION AND CONSOLIDATION: METHODOLOGICAL RESEARCH ON FOCUS GROUPS Research Question and Design

Consolidation Another data-integration strategy is consolidation. Compared with data transformation, consolidation goes one step further in terms of data integration. For data consolidation, a quantitative data set is not augmented solely by the addition of converted qualitative data, but rather new variables are created through the merging of qualitative and quantitative data. Consolidating (or merging) data leads to new data sets in either qualitative or quantitative form for further analysis (Caracelli & Greene, 1993). Thus, data from different sources have to take the same (or similar) format, which means that some form of data transformation is required (Collingridge, 2012; Sandelowski et al., 2009; van Velzen, 2016). Within a consolidated data set, distinguishing between the originally separate qualitative or quantitative data is not possible; instead, a genuinely new piece of information is generated (Bazeley, 2018b; Caracelli & Greene, 1993; Creswell & Plano Clark, 2018). Consolidating data sources is a generative strategy that is often used to resolve puzzles raised by the data, but it can also raise new questions (Bazeley, 2018b). Consolidated datasets often take a quantitative form—that is, qualitative data are transformed into numbers, and these numbers, paired with the quantitative data, constitute a new data set (e.g., Jang et  al., 2008). However, data transformation can go both ways—quantitizing and qualitizing (Bazeley, 2018b; Vogl, 2017, 2019). For example, case profiles can be created from different data. They are “written into a consolidated form in which text is interspersed with or complemented by visual, numeric, and/or statistical data. (…) Assuming there are multiple cases in the study, these integrated compilations or narratives then become the basis for further analysis” (Bazeley, 2018b, p. 139). In the next section, I will illustrate an integrated analysis based on a transformation

In a methodological research project, I studied specificities and general applicability of focus groups with children aged 6 to 15 years (Vogl, 2019). The analysis dealt with formal aspects, such as the progress of the conversation, group dynamics, role differentiation among participants, and participants’ abilities. The raw data stemmed from “qualitative” data collection—namely, five age-homogeneous focus groups. However, these focus groups and the video recordings thereof contained a wide range of different types of information. Initially, I analyzed the data in three different strands independently: a qualitative, a quantitative, and a transformation strand.

Initial Separate Analysis In the qualitative strand, utterances were analyzed in terms of their type of content—for example, whether an utterance was a new aspect or a repetition or confirmation of something said before. For example, analyzing the contribution to a task that aimed for consensus, I found that 6- to 7-year-olds rarely referred to other participants’ utterances and instead repeated the same suggestions and ideas. Thus, neither compromises nor consensus were established. In contrast, from about 10 years of age, more mutual references were made. Quantitative characteristics were word counts, number of “private” conversations in a subgroup, duration of pauses, and number of overlapping and of incomprehensible utterances. This data showed that the 6- and 7-year-olds had the highest share of overlapping utterances, which also led to a high share of incomprehensible sections and a low share in pauses. On the one hand, this result illustrates the high engagement of the youngest focus group participants, but on the other hand, it

148

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

also shows the practical challenges in conducting (and transcribing) focus groups. In the transformation strand, I coded utterances with the Interaction Process Analysis (IPA) coding scheme developed by Robert Bales for small group analysis (1972). This scheme consists of 12 categories—six for positive and negative socialemotional behaviours and six for task-related behaviours (questions and answers). Subsequently, interaction profiles of participants were created with information on frequencies of these 12 types of interactions. Furthermore, interaction matrices —so-called who-to-whom matrices—with information on direction and type of interaction could be constructed. Based on these matrices, I created visual representations of interaction frequency and direction of interactions in a simplified network graph, which illustrated the relations in each group of participants and facilitated age comparisons. Subgroups became evident, as did the involvement of individual participants and the centrality of the moderator. For example, the interaction profile of the focus group with the first graders (6–7 years) showed a balance between positive and negative social-emotional behaviour. The comparatively high level of disagreement illustrated little pressure for conformity, which clearly changed with the increasing age of the participants.

Data Integration in Practice Within these three strands, a group-level and an individual-level analysis can be distinguished. On the group level, aggregated information was integrated to focus group profiles. I linked results from the qualitative, quantitative and transformation strands into profiles per age group with a focus on participation and motivation, and on collaboration and interaction. Original strands could still be identified in this step, although they were integrated in a narrative description of the age groups. The following segment illustrates the results for one age group and the link to the respective strand of analysis: The 10- and 11-year-olds had an enormous need to talk (QUAL + QUAN) and were quite excited (QUAL). No reservation or nervousness was noticeable (QUAL + QUAN). The approval and sympathy of the moderator frequently seemed to be more important to the participants than their contribution to the content of the discussion (QUAL). Thus, the moderator was the main addressee of interactions (TRANS). At the same time, interaction among peers remained limited (TRANS). (Vogl, 2019)

In the next step, data was integrated through data linkage, transformation and consolidation. On the individual level, information was compiled to a participant profile for each participant. For this step, the qualitative, quantitative, and transformed information was consolidated by integrating all results from the three strands to verbal (or qualitative) profiles. With this consolidation into individual role profiles per participant, age-specific types of behaviour and underlying cognitive, social, and verbal skills could be described. The results were case profiles as a new form of data. The following section illustrates this data consolidation and how, for many aspects, data are integrated to such a degree that the original strand cannot be identified any more: Participants aged 6 and 7 showed great differences in their behavior. Some were rather quiet, others rather loud. I could identify two leaders. One could be described as class clown and showed inappropriate behavior by disturbing the focus group by suddenly laughing out loud, starting to sing, or dropping to the floor a couple of times. At the same time, this participant could be described as the opinion seeker who prods the group to contribute new aspects. The other leader could be considered the opinion giver by imposing personal views onto others and by being ruthless and aggressive at times. These two examples show that the two most active participants took self-centered roles, which was problematic for both the group maintenance and the content of the discussion. (Vogl, 2019)

In a last step, these case profiles were compared both within and across age groups to draw conclusions about methodological specificities of focus groups with children. The linked and consolidated data were used in a comparative analysis on the functioning of focus group discussions in different age groups and consequently facilitated metainferences on age-related specificities.

Reflections Reflecting on the analytic process and the results, I conclude that the Interaction Process Analysis in the transformation strand was the core of the study, but participants’ role descriptions would have been too superficial to answer the research question without additional qualitative analysis. Without the additional quantitative aspects, age comparisons and the characterization of the course of the discussion would have been far more difficult. Furthermore, the number of words alone did

DATA INTEGRATION AS A FORM OF INTEGRATED MIXED ANALYSIS

not provide information on verbal or cognitive skills. Qualitative-descriptive aspects were crucial to interpret the quantified information and gain a more detailed picture of verbal and communicative skills for children in focus group settings. In this mixed analysis, qualitative, quantitative, and transformed information provided insights into partly overlapping but generally distinct aspects of the research problem. The linked and consolidated data offered yet another dimension of results. Linking components of complementary data offered a more nuanced account of how the different data corroborate, illustrate, or elaborate each other (Bazeley, 2018a). Integrating different aspects paid tribute to the multidimensionality of the data produced in focus groups. At the same time, the extensive raw data became manageable without neglecting either the breadth or depth, and the systematic comparisons between age groups regarding structure and content of focus groups were facilitated. Furthermore, through the merging of different types of information and creating a new qualitative data set, credibility and intersubjectivity of results could be extended, thereby increasing the reach of results. This study illustrates how the researcher’s perspective, not the data-collection methods itself, determines which data types to include and that data are constructed rather than given. It also demonstrates the added value as well as the complexity of integrated data analysis. With these different types of data, methods and techniques, different facets of the phenomenon could be understood. They were at the hub of the research enterprise; they mediated what could be learnt about the phenomenon.

Taking it Further: Data Types and the Problem of Q-Labels The example of researching the applicability of focus groups illustrates how one data-collection method cannot be classified as qualitative or quantitative per se: it naturally consists of different facets and offers different types of data. The deciding factor is what researchers (choose to) look at. What do they consider (relevant) data? Is it the “amount” of something or the content? This also means that data are always selectively constructed by means of the particular conceptual “lens” used by the researcher (Maxwell, 2010). Surrounding phenomena are ontologically and epistemologically neutral, and have both qualities and quantities. Thus, “qualitative and quantitative data are not so much different kinds of data as

149

these data are experiences formed into, for example, words or numbers, respectively” (Sandelowski et  al., 2009, p. 209). Whether we use numbers, words, or images does not change the phenomenon being described or analyzed, although it can change how we think about the phenomenon and how we present it to others (Bazeley, 2018a). Thus, the division into qualitative and quantitative data is not as straightforward as it may seem at first glance. To take this further: if data cannot be easily categorized as qualitative and quantitative, the separation of qualitative and quantitative methods is also questionable. As we saw in the example of focus group research, many methods can produce either quantitative or qualitative data (Poth, 2019). Which logic to research is taken depends on the approach to data collection and analysis, and not the method as such. The distinction between qualitative and quantitative research is only a “straw man” (Bergman, 2008) relating to a considerable degree to “delineating and preserving identities and ideologies rather than to describe possibilities and limits of a rather heterogeneous group of data collection and analysis techniques” (Bergman, 2008, p. 19). Thus, the qualitative–quantitative distinction is unproductive (Miles & Huberman, 2019) and does not describe different types of data or methods appropriately, but instead depicts certain mental models (Maxwell, 2010). Furthermore, the dualism does not help in conceptualizing (or describing) mixed methods analysis. Verbal utterances would be described as qualitative data and the number of words or word share as quantitative data. However, are word counts the product of data transformation, or do they exist independently? Are we quantitizing when we count words or interactions? Yes, we are, but in the same way, we would quantitize verbal utterances in a survey interview by assigning a number to the response. If quantitizing is about converting verbal data into numerical data, then survey interviews quantitize in a similar way as word counts or coding in qualitative analysis. The implication is that data transformation suggests that the material underlying research can be categorized as qualitative or quantitative. I would argue that the same material can be both and that it does not require a data transformation process in a strict sense. The transformation is or can be part of the data collection (recording) rather than the analysis. Thus, if the distinction between qualitative and quantitative data is blurred, the terminology for integrated data (and mixed methods in general) has limits. We should be aware of the limited value of the dichotomy in methods and data, and consequently of definitory complications.

150

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

The point is not to pick tools from a methodological toolbox with the labels “qualitative” and “quantitative”, but to select the best approach to answer a substantive research question. We integrate different types of data to integrate different ways of thinking about phenomena (Fielding, 2012) and not simply to apply methodological terms. As Lieber and Weisner (2010, p. 564) noted: “Rather than thinking about quantitative and qualitative data as separate pieces of the puzzle to be independently collected and analyzed, a more integrated perspective suggests a focus on the nature of the problem being addressed—the primary research question.” To conclude, despite the criticism on the qualitative–quantitative dichotomy, inventing new terms would not resolve the practical issue of how to best answer a research question. But terminology should not stifle creativity in integrated mixed analysis because this is what constitutes the added value of integration and mixed methods research more generally. Many times integrated analysis is evolving, exploratory, iterative, and requires a “spirit of adventure” (Greene, 2007, p. 144). On the downside, integrative analysis requires a breadth of skills as well as imagination, and the ability to envision what might be possible with inventiveness. On the upside, it can open new layers of understanding and explanation.

CONCLUSION: DATA IS AT THE HUB OF INTEGRATED ANALYSIS Integration is the hallmark of MMR and data integration the new gold standard in judging the quality of MMR. Integration can be achieved in different or multiple steps of the research process, with data analysis probably being the most challenging. Shedding light on the role of data in mixed methods analyses and integration is crucial. Data represent the “main act” in empirical research and can play various parts in mixed analysis. Integrated analysis makes different data types “talk to each other” (Moseholm & Fetters, 2017). This conversation can be dynamic and multidirectional, as well as agreeable or controversial (Kutscher & Howard, 2021). As a consequence, integrated analysis can be emergent and iterative, in contrast to planned and linear. This also implies that the purposes of mixing can develop or change during the research process (Schoonenboom et al., 2018). For a further discussion of visualizing the interactions in mixed methods, see also Chapter 8 (this volume).

In this chapter, the focus lies on (a) integrated analysis, in which analytic approaches based on different types of data interact, and (b) integrated data, which implies that data types are linked or transformed in such a way that they can be analyzed together or be used to build a new data set for further analysis (consolidation). In more detail, I described data transformation and consolidation as integrative strategies, and illustrated these strategies with an empirical example. The example shows how one data collection yields a wealth of different information. It also demonstrates how the researcher’s perspective, not the data-collection methods themselves, determines which data types to include and that data are constructed rather than given. Furthermore, the qualitative–quantitative distinction between data is blurred. The research problem determines what and how to analyze, but not the methodological terminology or conviction. Thinking outside the q-boxes can advance integration in data analysis, as well as the overall design. A less strict interpretation of and adherence to typologies can advance creativity and innovation in MMR, and advance the field in both substantive and methodological ways. This is not to say that integrated analysis and data integration are sacrosanct. It is neither always necessary nor always possible, and it comes with some problems. In particular, resources – human and financial – are obviously increased. Furthermore, quantitizing can suggest greater precision than intended, and the generalizability of results, as well as the applicability of statistical methods, can be limited owing to small numbers of cases and nonrandom sampling. Additionally, evidence should not be reduced to the amount of evidence available: “Numbers can’t replace the actual description of evidence but can provide a supplementary type of support for the conclusions” (Maxwell, 2010, p. 479). Critical reflection is crucial: Why and how are we integrating and to what end? What are compromises we (have to) make? What are legitimate claims? For the future, we need enhanced conceptual frameworks for and practical examples of integrated analyses with strong methodological reflections and a more critical view on typologies and terminologies. After all, integration is not a terminology problem, but a practical one. In that vein, it might also be advisable to open up the discourse beyond self-identified MMR. Integration does not necessarily mean the merging of qualitative and quantitative data, but rather the joining of any type of data or perspective (Vogl et al., 2019). The debate on integration in mixed method analysis can benefit from examples of integration in other fields. Thus, strict application of terminologies

DATA INTEGRATION AS A FORM OF INTEGRATED MIXED ANALYSIS

limits the debate about and progress of integration. There is not necessarily a need for one single, shared terminology. Although clear definitions can provide a solid conceptual framework and are a sign for the professionalization of a field, we need to keep our general frameworks somewhat flexible to allow for “informed creativity” (Mertens et  al., 2016, p. 3). Innovative and creative thinking around integrated data analyses and the publication of good-practice examples thereof could significantly advance the field and should not be stifled by terminology or q-boxes. Be flexible—let the research question inspire you, but do not get carried away and lost in your data and options for integration; stick to the research problem as a guiding light. Analytic approaches must have a clear relation to the research objectives. This does not necessarily imply that the research question or problem dictates the methods (Teddlie & Tashakkori, 2009). Research questions can also have a reciprocal relationship with methods: research questions shape methods, but they can also be shaped by methods (Mertens et al., 2016). Methods are not employed for their own sake but to understand a phenomenon. However, the facets of a phenomenon we can understand depend on methods and techniques. Data are at the hub of this interrelationship.

WHAT TO READ NEXT Bazeley, P. (2018). Integrating analysis in mixed methods research. Sage.

This book is a great resource with insights into the foundations, practical strategies and challenges in integrated analysis illustrated with plenty of inspiring examples. It offers a comprehensive, thorough, cutting-edge contribution and is invaluable for all mixed methods researchers. Caracelli, V. J., & Greene, J. C. (1993). Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 15(2), 195–207. https://doi.org/10.3102/01623737015002195

As one of the first articles with a focus on MMR data analysis, Caracelli and Greene identify four analytical strategies for integrating quantitative and qualitative data: transformation, typology development, extreme case analysis, and consolidation/merging. In this seminal work, they advanced the understanding of integrative data analysis strategies, which is still an important reference point.

151

Poth, C. N. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage.

In an innovative and pedagogical format, this book offers valuable guidance for mixed methods researchers. Poth convincingly discusses demands of complexity in mixed methods research and suggests six adaptive practices for integrative thinking with sensitivity to complexity, facilitating recognizing and making mindful decisions under conditions of complexity. Hereby, she lays the ground for future debates on integrating complexity and its effects in and on MMR.

REFERENCES Akremi, L., Baur, N., Knoblauch, H., & Traue, B. (Eds.) (2018). Grundlagentexte Methoden. Handbuch: Interpretativ forschen (1st ed.). Beltz Juventa. Bales, R. F. (1972). Interaction process analysis. In D. L. Sills (Ed.), International encyclopedia of the social science (pp. 465–470). Crowell Collier and Macmillan. Baur, N., & Knoblauch, H. (2018). Zur Interpretativität des Quantitativen: oder: Zur Kongruenz von qualitativer und quantitativer empirischer Sozialforschung. Soziologie, 47(4), 439–461. Bazeley, P. (2009). Editorial: Integrating data analyses in mixed methods research. Journal of Mixed Methods Research, 3(3), 203–207. https://doi. org/10.1177/1558689809334443 Bazeley, P. (2010). Computer-assisted integration of mixed methods data sources and analyses. In A. Tashakkori & C. Teddlie (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 431–467). Sage. Bazeley, P. (2012). Integrative analysis strategies for mixed data sources. American Behavioral Scientist, 56(6), 814–828. https://doi.org/10.1177/00027642 11426330 Bazeley, P. (2016). Mixed or merged? Integration as the real challenge for mixed methods. Qualitative Research in Organizations and Management: An International Journal, 11(3), 189–194. https://doi. org/10.1108/QROM-04-2016-1373 Bazeley, P. (2018a). “Mixed methods in my bones”: Transcending the qualitative-quantitative divide. International Journal of Multiple Research Approaches, 10(1), 334–341. https://doi.org/10. 29034/ijmra.v10n1a22 Bazeley, P. (2018b). Integrating analysis in mixed methods research. Sage. Bergman, M. M. (2008). The straw men of the qualitative-quantitative divide and their influence on

152

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

mixed methods research. In M. M. Bergman (Ed.), Advances in mixed methods research: Theories and applications (pp. 11–21). Sage. Blasius, J., & Friedrichs, J. (2008). Lifestyles in distressed neighborhoods: A test of Bourdieu’s “taste of necessity” hypothesis. Poetics, 36(1), 24–44. https://doi.org/10.1016/j.poetic.2007.12.001 Brannen, J., & O’Connell, R. (2015). Data analysis I: Overview of data analysis strategies. In S. N. Hesse-Biber & B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 257–274). Oxford University Press. Bryman, A. (2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1(1), 8–22. https://doi.org/10. 1177/2345678906290531 Caracelli, V. J., & Greene, J. C. (1993). Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 15(2), 195–207. https://doi.org/10.3102/01623737015002195 Collingridge, D. S. (2012). A primer on quantitized data analysis and permutation testing. Journal of Mixed Methods Research, 7(1), 81–97. https://doi. org/10.1177/1558689812454457 Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Fakis, A., Hilliam, R., Stoneley, H., & Townend, M. (2014). Quantitative analysis of qualitative information from interviews: A systematic literature review. Journal of Mixed Methods Research, 8(2), 139–161. https://doi.org/10.1177/1558689813495111 Fetters, M. D., & Freshwater, D. (2015). The 1 + 1 = 3 integration challenge. Journal of Mixed Methods Research, 9(2), 115–117. https://doi.org/10.1177/ 1558689815581222 Fielding, N., & Fielding, J. L. (1986). Linking data. Sage. Fielding, N. G. (2012). Triangulation and mixed methods designs. Journal of Mixed Methods Research, 6(2), 124–136. https://doi.org/10.1177/1558689 812437101 Greene, J. C. (2007). Mixed methods in social inquiry (1st ed.). Jossey-Bass. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. Guetterman, T. C., Molina-Azorin, J. F., & Fetters, M. D. (2020). Virtual special issue on “integration in mixed methods research.” Journal of Mixed Methods Research, 14(4), 430–435. https://doi.org/ 10.1177/1558689820956401 Jang, E. E., McDougall, D. E., Pollon, D., Herbert, M., & Russell, P. (2008). Integrative mixed methods data analytic strategies in research on school success in challenging circumstances. Journal of

Mixed Methods Research, 2(3), 221–247. https:// doi.org/10.1177/1558689808315323 Johnson, R. E., Grove, A. L., & Clarke, A. (2017). Pillar integration process: A joint display technique to integrate data in mixed methods research. Journal of Mixed Methods Research, 13(1), 301–320. https://doi.org/10.1177/1558689817743108 Kuckartz, U. (2017). Datenanalyse in der mixedmethods-forschung. Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(2), 157–183. https:// doi.org/10.1007/s11577-017-0456-z Kutscher, E. L., & Howard, L. C. (2021). Integration as a process: Applying iterative multiple correspondence analysis to surface dynamic findings. Journal of Mixed Methods Research, 1558689 82110216. https://doi.org/10.1177/155868982110 21669 Li, S., Marquart, J. M., & Zercher, C. (2000). Conceptual issues and analytic strategies in mixed-method studies of preschool inclusion. Journal of Early Intervention, 23(2), 116–132. Lieber, E., & Weisner, T. S. (2010). Meeting the practical challenges of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 559–579). Sage. Maxwell, J. A. (2010). Using numbers in qualitative research. Qualitative Inquiry, 16(6), 475–482. https://doi.org/10.1177/1077800410364740 Mertens, D. M., Bazeley, P., Bowleg, L., Fielding, N., Maxwell, J. A., Molina-Azorin, J. F., & Niglas, K. (2016). The future of mixed methods: A five year projection to 2020. https://mmira.wildapricot. org/resources/Documents/MMIRA%20task%20 force%20report%20Jan2016%20final.pdf Miles, M. B., & Huberman, A. M. (2019). Qualitative data analysis: An expanded sourcebook (4th ed.). Sage. Moran-Ellis, J., Alexander, V. D., Cronin, A., Dickinson, M., Fielding, J., Sleney, J., & Thomas, H. (2006). Triangulation and integration: Processes, claims and implications. Qualitative Research, 6(1), 45–59. https://doi.org/10.1177/1468794106 058870 Moseholm, E., & Fetters, M. D. (2017). Conceptual models to guide integration during analysis in convergent mixed methods studies. Methodological Innovations, 10(2), 1–11. https://doi. org/10.1177/2059799117703118 Niglas, K., Kaipainen, M., & Kippar, J. (2008). Multiperspective exploration as a tool for mixed methods research. In M. M. Bergman (Ed.), Advances in mixed methods research: Theories and applications (pp. 172–188). Sage. Onwuegbuzie, A. J. (2003). Effect sizes in qualitative research: A prolegomenon. Quality & Quantity, 37, 393–409.

DATA INTEGRATION AS A FORM OF INTEGRATED MIXED ANALYSIS

Onwuegbuzie, A. J., & Combs, J. P. (2010). Emergent data analysis techniques in mixed methods research: A synthesis. In A. Tashakkori & C. Teddlie (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 397–430). Sage. Onwuegbuzie, A. J., & Hitchcock, J. H. (2015). Advanced mixed analysis approach. In S. N. HesseBiber & B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 275–295). Oxford University Press. Onwuegbuzie, A. J., & Hitchcock, J. H. (2019). Using mathematical formulae as proof for integrating mixed methods research and multiple methods research approaches: A call for multi-mixed methods and meta-methods in a mixed research 2.0 era. International Journal of Multiple Research Approaches, 11(3), 213–234. https://doi. org/10.29034/ijmra.v11n3editorial2 Poth, C. N. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Sandelowski, M. (2000). Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed-method studies. Research in Nursing & Health, 23, 246–255. Sandelowski, M., Voils, C. I., & Knafl, G. (2009). On quantitizing. Journal of Mixed Methods Research, 3(3), 208–222. https://doi.org/10.1177/15586898 09334210 Schoonenboom, J., & Johnson, R. B. (2017). How to construct a mixed methods research design. KZfSS

153

Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69, 107–131. https://doi.org/10.1007/ s11577-017-0454-1 Teddlie, C., & Tashakkori, A. (Eds.) (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage. van Velzen, J. H. (2016). Students’ general knowledge of the learning process: A mixed methods study illustrating integrated data collection and data consolidation. Journal of Mixed Methods Research. Advance online publication. https://doi. org/10.1177/1558689816651792 Vogl, S. (2017). Quantifizierung: Datentransformation von qualitativen Daten in quantitative Daten in Mixed-Methods-Studien. Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 69(2), 287– 312. https://doi.org/10.1007/s11577-017-0461-2 Vogl, S. (2019). Integrating and consolidating data in mixed methods data analysis: Examples from focus group data with children. Journal of Mixed Methods Research, 13(4), 536–554. https://doi. org/10.1177/1558689818796364 Vogl, S., Schmidt, E.-M., & Zartler, U. (2019). Triangulating perspectives: Ontology and epistemology in the analysis of qualitative multiple perspective interviews. International Journal of Social Research Methodology, 22(6), 611–624. https://doi.org/10. 1080/13645579.2019.1630901 Yin, R. K. (2006). Mixed methods research: Are the methods genuinely integrated or merely parallel. Research in the Schools, 13(1), 41–47.

11 Ethical Issues and Practices for Mixed Methods Research in an Era of Big Data Roslyn Cameron and Heinz Herrmann

INTRODUCTION This chapter focuses on ethics in MMR related to big data and ways to address ethical dimensions specific (or even unique) to designing MMR studies that include big data. The MMR community has been concerned with ethics before the advent of the mass utility of big data. However, the use of big data in MMR produces new ethical issues. Nonetheless, the need for methodological transparency and the challenges MM researchers face in doing this remain a central consideration. Many mixed methodologists argue that ethical issues need to be considered across the entire research process (Cain et al., 2019; Collins et al., 2012; Hesse-Biber, 2010; Hesse-Biber & Johnson, 2013; Hesse-Biber & Leavy, 2006; Onwuegbuzie & Corrigan, 2014; Poth, 2018; Preissle et  al., 2015), and this is particularly pertinent to mixed and complex research designs where ethical issues can be emergent and reflective of the interplay between both qualitative and quantitative data collection. Transparency of methodological decisionmaking processes in MMR is seen as crucial (Cain et  al., 2019; Collins et  al., 2013; Preissle et  al., 2015), with particular care taken for the fluidity and contingent nature of MMR designs and processes, and the unexpected and often amplified ethical issues that can arise (Preissle et al., 2015).

Generally, ethical codes refer to a set of ethical principles and issues related to research with human participants such as: informed consent, privacy, confidentiality, anonymity, minimizing any possibility of deception and any risk of harm (physical or psychological) to research participants. Special ethical risk mitigations and sensitivities are also in place in these codes for vulnerable groups—for example, children (those under 18 years of age), members of ethnic minorities, people with disabilities and Indigenous peoples. Poth (2018) posits three overarching principles for ethical research with human participants in relation to MMR: respect for persons (consensual arrangements and processes); concern for welfare (confidentiality, privacy, use of incentives and dialogue); and concern for justice (sampling and sampling criteria, recruitment of participants). Preissle et  al. (2015) in their work on ethics for MMR refer to six major ethical principles: “fairness, duty, consequences, reasonableness, virtuousness, sociability and caring” (p. 146), while Bell and Wray-Bliss (2010) argue for organizational researchers to develop “dispositions such as honesty, sensitivity, respectfulness, reciprocity and reflexivity” (p. 89) alongside the more formal compliance approaches to ethical research, as thematized by Cain et al. (2019) in their research on ethics and reflexivity in MMR and referred

ETHICAL ISSUES AND PRACTICES FOR MIXED METHODS RESEARCH

to by Preissle et al. (2015) as “compliance”: and “research integrity” approaches to ethical research. In MMR, ethics is related to reflexivity, because aspects of reflexivity-related decision making is increasingly being seen as a methodological consideration and a strategy for minimizing bias in the research process. Cain et al. (2019) examined ethics and the use of reflexivity in published MMR through a modified systematic review in combination with simple random sampling, resulting in a selection of studies (n = 322) in the ERIC database for a five-year period (2013–2018). Definitions of ethics and perspectives on ethics were identified in their review of ethics and reflexivity in MMR: “Ethics as defined by an Institutional Review Board (IRB), Data Quality as a Measure of Ethics; Ethics as Defined by Theory, and Social-JusticeMinded ethics” (Cain et al., 2019, p. 144). A further three themes were identified in the systematic review of papers: researcher positionality, reflexivity as a methodological consideration and minimizing bias. They argue that: describing ethics and reflexivity are of the utmost importance because high-quality methodological practice requires reflexivity and demonstration of ethical inquiry to establish credibility and legitimation. We also argue that transparency around ethical- and reflexivity-related decision making can strengthen writing and promote clarity when dealing with typically complex design. (Cain et  al., 2019, p. 144)

Preissle et al. (2015), in their exploration of ethics for MMR, focus on five key ethical research decision coordinates for MMR. These include ethics of research design and purpose; ethics of MMR sampling and selection; ethics of research relationships; ethics of data collection and analysis; and ethics of representation and reporting. Consistent with the findings from Cain et  al. (2019), reflexivity and transparency are viewed as central to ethical decision-making in MMR. Hesse-Biber and Griffin (2013) examined the use of internet-mediated technologies in MMR and the ethical issues that emerge from this. They address new implications for this through medium theory and acknowledge “online modes of research change the nature of discourse” (p. 58). Stadnick et  al. (2021) undertook an empirical study to examine ethics as it pertains to health services MMR. The study highlighted unique challenges faced by MM researchers in this field of inquiry, and identified the ethical issues most frequently encountered and those most perceived to be difficult to mitigate. This study concludes by advocating for specialized MMR training programmes that focus on ethical integrity for MMR

155

health services research. Similarly, Hesse-Biber and Griffin (2013) suggest the need for development of social science researchers to continually upskill and to become cognizant of the ethical implications of applications of internet-mediated technologies in MMR. The increasing use of big data and associated technologies and intelligences in MMR is the prime focus of this chapter along with the associated ethical issues that arise from this increasing utility. These technologies and analytical capabilities have been advanced by rapid and vast improvements in statistical and computational power, especially in relation to big data, cloud storage and the Internet of Things (IoTs). It is anticipated that the use of big data in MMR will increase and make the need for more thorough and informed ethical decision-making in MMR. It has become imperative that those engaging in MMR design need to be explicitly and proactively aware of, and cognisant of, the ethical implications of utilising, collecting and analysing big data and associated technologies across all these methodological decision-making coordinates proposed by Preissle et al. (2015). The attention of this chapter now turns to the emergent but expanding use of big data and associated digital technologies in MMR and associated ethical AI principles. First, we present a conceptual framework that depicts the technology relationships in big data before providing comprehensive definitions of key terms. This is followed by a discussion on ethical AI principles: beneficence, nonmalevolence, justice, explicability and autonomy. Examples of the use of big data in MMR studies are presented and then a more detailed description of two of these studies follows before concluding the chapter.

ETHICAL ISSUES ARISING FROM THE USE OF BIG DATA IN MMR A Framework for Big Data A framework for big data inclusive of its conceptual components is presented below as a Venn diagram in Figure 11.1, based on a taxonomy developed by Herrmann (2022). Below, definitions for these key conceptual components are presented, along with example applications in MMR. For a further discussion of big data in MMR, see also Chapter 10 (this volume). Machine learning examines the ways in which computers—“machines”—can learn. It takes place through an entirely data-based algorithm,

156

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 11.1  Big data components relevant for MMR researchers Source: Authors.

which creates a “conversation” (the learning part of machine learning) between input and output data, optimizing modelling parameters until the input predicts the output well (Herrmann & Masawi, 2022). The resulting (or trained) model with its optimized parameters then becomes the AI code for use by researchers or data scientists in industry (Schlenker & Minhaj, 2020). Machine learning requires a scientist to have a solid grounding in linear algebra, calculus and probability theory (Serrao, 2020). An example of MMR with machine learning is provided by Hillard et al. (2008) for classifying documents in the social sciences. First, annotators grouped a subset of documents into various classes. Next, this classified subset was used by the researchers to train predictive models with various machine learning algorithms. The remaining bulk of the documents were then classified with these models to perform a pattern analysis, which was linked with caselevel investigations. Deep learning is a sub-area of machine learning that relies on neural network technology, which emulates the neurons and synapses of the human brain (Leijnen & Veen, 2020). Such neural networks have several internal layers and become more “intelligent” the more layers they have (Borges et al., 2021). The strength of deep learning is in its ability to deal effectively with unstructured data, such as in natural language processing, computer vision and big data (Nelson, 2021). An example of deep learning in MMR is provided by Wiguna et al. (2020), who use deep learning as a diagnostic tool for attention-deficit/hyperactivity disorder (ADHD) in a psychiatric MMR project. Natural language processing teaches computers to understand text and voice data. Machine

learning has been employed in natural language processing since the 1980s (Nadkarni et al., 2011) and is increasingly used in combination with deep learning for applications in sentiment and content analysis, language translation, word sense disambiguation, summarization, syntactic annotation or named entity recognition (Zeroual & Lakhouaja, 2018). A productized example for researchers is the Leximancer tool, which uses machine learning for semi-automated analytics in the qualitative data synthesis (Nunez-Mir et al., 2016). Computer vision endows computers with the ability to interpret digital images, videos and other visual inputs. Typical applications are image classification (i.e., what objects are in an image such as a face), object detection (i.e., photo tagging of a particular person’s face or where objects are in an image) and image segmentation (i.e., how objects in an image are related to analyze a scene [Feng et al., 2019]). Similar to natural language processing, the fusion of machine and deep learning has become the dominant technology for computer vision (O’Mahony et al., 2020). Specialized hardware is used for the deep learning component, such as graphical or tensor processing units (Herrmann, 2022). One example of computer vision in MMR is O’Halloran et al. (2019), who provide an automated analysis of text and image relations in their MMR study on violence and extremism. Big data is the most recent AI technology. It relates to more than just large data sets and includes technologies to process the data, such as machine and deep learning, cloud computing and the internet of things. Big data is conceptualized in terms of four Vs: Variety (data from database systems and spreadsheets, natural language processing and computer vision); velocity (the rate

ETHICAL ISSUES AND PRACTICES FOR MIXED METHODS RESEARCH

of new data being generated and its processing requirements); volume (sometimes in the order of petabytes magnitude of data); and veracity, which involves dealing with data errors, missing data or outdated/obsolete data (Arthur & Owen, 2019; Nicholls et al., 2016). Often a fifth V is added for “value” and nowadays referred to as “visualization,”, which relates to the meaningful interpretation of big data into findings (De Mauro et al., 2016). This interpretation can be conducted in real-time (i.e., “live” and in synchronization with the data being generated) or as a delayed batch processing of historical data. When real-time analysis of big data is conducted, this is often referred to as “advanced analytics” (Sena et al., 2019). One example of delayed batch processing of big data in MMR is Andreotta et al. (2019), who used big data for quantitative topic alignment on Twitter, followed by a qualitative thematic analysis. Cloud computing: the volume and velocity of big data require expensive hardware infrastructure and software for accessing, storing and analyzing large datasets (Crespo-Perez & OjedaCastro, 2017). Increasingly, these resources have been “outsourced” to data centers in the internet “cloud”, which provides services for their use on demand (Chang, 2015). Commercial cloud computing services for big data, including visualizations, are offered by Amazon, Google, Microsoft, etc. (Sangeetha & Sreeja, 2015). Two examples of cloud computing in MMR are Dubois and Ford (2015), who integrated qualitative trace interviews with free cloud-based tools for summarizing textual data and discovering communication networks from social media posts. Moats and Borra (2018) developed their own tool for integrating qualitative and quantitative data from Twitter. The internet of things goes beyond social media and feeds big data with a wider range of sources, such as sensors, wearables, machines, actuators and vehicles (Berryhill et al., 2019). Together with cloud computing, the internet of things is a foundational technology of what is called the “Fourth Industrial Revolution” (4IR)—i.e., the use of industry 4.0 applications (Liao et al., 2018). Due to the real-time processing requirements of the internet of things (velocity), telecommunications bandwidth has become a bottleneck (Satyanarayanan et al., 2009). Two example studies of the internet of things in MMR are Müller et  al. (2019), who provide a conceptual paper on using sensors in MMR. An MMR example with lower real-time requirements is presented by Bornakke and Due (2018) using GPS trackers. Big data is a popular subject in MMR for its multimodal analysis of images, videos, words and audio—i.e., the “variety” aspect of big data (Mertens et al., 2016). The major components to

157

understand for MM researchers when conducting a multimodal analysis are machine and deep learning. This has caused the terms “machine learning”, “deep learning” and “big data” often to be conflated in MMR (Ratia et al., 2019), as will be discussed later in this chapter. However, the above discussion of Figure 11.1 shows that variety is only one aspect of big data, and researchers also need to have a good grounding in cloud computing and the internet of things in order to address the other Vs of big data—i.e., volume, velocity, veracity, and value. In other words, researchers need to understand the intricacies of how big data can be collected, stored, processed, and analyzed in real time. For one reason, there is ever more data (i.e., volume). According to IBM in 2016, 90 per cent of the volume of data in the world was created in the previous two years alone (Loechner, 2016). In addition, the velocity of data from social media and the internet of things exponentially creates more data and updates existing data, which often needs to be processed in milliseconds (i.e. real time) for meaningful interpretation in industry 4.0 applications (Arthur & Owen, 2019), unless the interpretation (value) is restricted to trends (Sangeetha & Sreeja, 2015). Data veracity (trustworthy data) also needs to be considered, which requires specialized skills due to the complex structure of large data sets, and their imprecision and inconsistency (Sivarajah et  al., 2017). However, research has shown that scientists mostly do not have skills across all big data components even if they have a strong quantitative foundation (Davenport, 2020; Nelson, 2021). For researchers with a qualitative background, “AutoML” tools are emerging to “democratize” at least the machine and deep learning techniques for a multimodal analysis of trends from historical data (Hurtgen et al., 2020). This facilitates the uptake of multimodal analysis by MM researchers but has ethical ramifications, as will be discussed next.

Ethical Aspects of Big Data and Implications for MMR In this chapter, we will discuss the following ethical principles and their relevance to big data in MMR: beneficence, non-maleficence, autonomy, justice and explicability. This choice is not without problems, though. Although more than 160 documents have been released about “ethical AI” principles from industry, government, institutions and NFP organizations (Jaume-Palasí et al., 2021), a recent study of the 24 most referred principles has shown that the conceptualization of principles

158

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

varies substantially across these documents and is dominated by wealthy Western countries from the Global North (Yampolskiy, 2013). This presents a problem for the operationalization of principles, because the underlying values of principles are not universal across geographical areas (Awad et al., 2018). For the purpose of providing a reference guide, we briefly describe five universal ethical AI principles. Four of them were derived from the field of bioethics (beneficence, non-maleficence autonomy and justice) with explicability added as an additional principle for an epistemological, as well as an ethical perspective in terms of accountability (Floridi & Cowls, 2019). • Beneficence relates to the promotion of human rights and well-being (Floridi et al., 2020). Privacy is an important component of beneficence (Herschel & Miori, 2017) and several privacyprotecting technologies have been proposed for big data, which need to be understood by MM researchers (Russell & Norvig, 2020). For example, Facebook settled for $650m in one of the largest privacy-related abatements to date (Moyer, 2021). • Non-malevolence avoids causing harm to people, such as cybercrime, fake news and radicalization of groups in our society. Published ethical AI frameworks are currently more focused on nonmalevolence than beneficence (Herrmann, forthcoming 2023). For MM researchers, this implies taking precautionary measures to prevent their research data from being “hacked” into. • Justice is a broader term for advocating the use of AI to rectify bias and inequalities in gender, race, social status, sexuality, disabilities or vulnerable populations (Guszcza et al., 2021). Most of the overall debate in ethical AI currently revolves around bias (Blackman, 2021) with implications for MM researchers to examine their training data for potential bias (Fitsch et al., 2020). • Explicability requires big data algorithms to become interpretable in human language because their internal technical workings are a “black box” that cannot readily be interpreted (Guszcza et  al., 2021). Ultimately, explicability enables researchers to be held accountable for unintended societal consequences (European Union, 2020). MM researchers need to consider explicability at the MM design stage when using big data (Martin, 2019). • Autonomy promotes human autonomy over AI autonomy. To relate to the risk of neglecting this

principle, we point to sci-fi movies such as the The Terminator or 2001: A Space Odyssey. Some experts predict that AI will exceed human intelligence in the next 30 years (Müller & Bostrom, 2016). This principle has more to do with the ethical development of AI for benefits to humankind than the ethical use of big data by researchers in MMR (Herrmann, forthcoming 2023). The growing importance of big data brings with it several ethical issues. Most of the ethical debate revolves around justice (Blackman, 2021). The appeal of big data democratization through AutoML becomes a problem when it turns into “algorithmic fetishism”, which may lead to correlations being interpreted as causations or when the veracity of the data is unquestioned. Algorithmic fetishism has been discussed by ethnographers (Thomas et al., 2018), researchers in business (Leicht-Deobald et al., 2019) and data feminists (D’Ignazio & Klein, 2020). Bias is a part of the equity principle (Guszcza et  al., 2021) and has various sources as it arises from non-representative data, bias inherent in representative data, choice of algorithms and human interpretations (Ashta & Herrmann, 2021). Therefore, any blindsided focus on a researcher’s “favorite” big data algorithm at the expense of examining training data for bias clashes with the justice principle. One way to counter bias in big data is to combine it with qualitative research, and thus perform mixed methods research. The term “thick data” has been proposed to complement big data with qualitative research to assist with the interpretation of results beyond visualization and advanced analytics (Thompson, 2019). The idea of thick data is to provide answers to both “what” and “why” questions to ensure the validity and ethical cogency of findings (Boyd, 2010). This is why thick data is sometimes referred to as “qualitative analytics” (Fiaidhi & Mohammed, 2019). For example, Bornakke and Due (2018) developed a multimethod framework for thick data to calibrate big data, contextualize big data, supplement “why” questions and add a behavioral scale. Originating from MMR in smart cities, Smets and Lievens (2018) developed a tool for thick data in ethnographic research.

Examples of Big Data utilized in MMR All this raises the question: To what extent has big data already been combined within MMR? To

ETHICAL ISSUES AND PRACTICES FOR MIXED METHODS RESEARCH

answer that question, we scanned the Scopus database as a starting point for finding MMR that includes big data. Scopus covers a wider range of journals than the Web of Science (Martín-Martín et  al., 2018) and the majority of articles on big data outside the Web of Science can be found in Scopus (Ruiz-Real et  al., 2020). A total of 77 documents were identified by the Scopus search with Boolean terms from the framework of Figure

159

11.1. However, when each article was examined more closely, it was found that only five related to empirical MMR research with big data. The remaining documents were either false positives or conceptual papers. An extended search in Google Scholar and the Google search engine provided one additional MMR study with empirical big data research. Table 11.1 provides an overview of these studies.

Table 11.1  MMR studies using big data in Scopus until 2021 (n = 6) Authors

Discipline

Research design: data collection and analysis

Andreotta et al. (2019)

Psychology

•  Integrates quantitative topic alignment and qualitative thematic analysis. •  Large Twitter dataset. •  Framework for compressing BD into smaller data sets for qualitative analysis. Convergent mixed methods design with qualitative to quantitative data tansformation

Holtrop et al. (2019)

Medicine

•  Integration of theory and data transformation procedures for large qualitative and quantitative data sets. TM in mixed methods designs (sequential and embedded)

Isoaho et al. (2021)

Policy science

•  Integration of quantitative topic modelling methods with qualitative methods for content and classification, and discourse and representation. •  Text mining is used to identify representative texts or examine their narrative structure before a qualitative method. Digital mixed methods design

O’Halloran et al. (2016)

Disaster and crisis management

•  Integrates qualitative methods of multimodal discourse analysis with quantitative methods of data mining. •  Visualization in a multilevel, contextual model for analyzing large data sets of multimodal texts. Mixed methods approach for big data analysis

O’Halloran et al. (2019)

Violence and extremism: political science

•  Integrates qualitative methods of multimodal discourse analysis with quantitative methods of data mining. •  Visualization with automated analysis of text and image relations. Case study: MMR design (data mining)

Poth et al. (2021)

Public health

•  N  ested convergent sequential MMR design informed by complex adaptive systems. •  Combines sentiment analysis in stage 1 with topic modelling in stage 2.

Mixed method four phased framework

Source: Authors.

160

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Discussion of Two Examples Next, two MMR studies from Table 11.1 will be examined in more detail regarding their consideration of big data components and ethical AI principles (as per the above section on ethical aspects of big data). Andreotta et al. (2019) and Poth et al. (2021) were selected based on their transparency and level of detail with respect to these aspects, and are summarized in Tables 11.2 and 11.3. Andreotta et  al. (2019) develop an extraction process from large-scale social media data for qualitative analysis, which overcomes shortcomings of historical judgement sampling approaches to such extraction problems. The MMR design is in four stages: 1 QUAL: Sample of n = 210,506 Twitter tweets. 2 QUAN: Two parallel big data analytics leading to a subset of topics for n = 205 topics. 3 Quan: Frequency analysis leading to 22 tweet topics. 4 QUAL: Thematic analysis and meta-inference. The second research by Poth et al. (2021) uses a convergent sequential MMR design, informed by complexity theory. The study cites Bulut and Poth (2022) as a cross-reference for additional details

on the quantitative sentiment analysis in stage 1, leading to six substantial changes in 72 public health briefings on COVID-19 in Canada. The published article focuses on the following themes: 1 Communicating risk assessments and measures. 2 Demonstrating empathy and rapport-building. 3 Updating information and actions. Table 11.2 shows that both studies addressed big data’s variety aspect through a qualitative analysis of natural language data. Large data sets (volume) were processed from either Twitter or media briefings. The validity of the research was increased by addressing the veracity of the data through a cross-analysis by qualitative and quantitative techniques. Regarding velocity and value, neither of the two studies’ research questions from Table 11.2 required a real-time analysis of large data sets. Instead, they only required the analysis of data snapshots for a trend analysis (i.e., batch-processing of historical, large data sets). Indeed, this was the case for all studies in Table 11.1. Therefore, big data in MMR appears to be avoiding the velocity and value aspects of realtime big data analytics, which contrasts with big data deployments in industry (Sangeetha & Sreeja, 2015). For example, chatbots in customer service

Table 11.2  Consideration of big data components Research

Variety

Andreotta et al. (2019)

Poth et al. (2021)

Source: Authors.

Volume

Velocity

Veracity

Qualitative, 201,506 Twitter thematic tweets with analysis of 220 205 topics Twitter tweets that were quantitatively compressed

Batched analysis of CSIRO’s Emergency Situation Awareness (ESA) platform

Media briefing transcripts, COVID case statistics, key event timeline

Batched analysis of publicly available data

Non-negative Visualization, matrix tokenization inter-joint and statistics factorization modules (NMijF) in Python considers software, precontent and processing in socio-temporal MySQL relationships of tweets Qualitative Visualization, dominant tokenization, cross-over and statistics analysis, modules in R including topic software; word modelling, classification in qualitative Bing lexicon themes, timeline events, and descriptive statistics

72 media briefing transcripts

Value

ETHICAL ISSUES AND PRACTICES FOR MIXED METHODS RESEARCH

161

Table 11.3  Consideration of ethical principles for big data Research

Beneficence

Non-malevolence

Equity

Andreotta et al. (2019) Privacy was Malevolent themes Reduction of bias maintained by were identified in the topic anonymizing user among other representation names and links to results, such through online websites as denying the survey with 154 legitimacy of respondents that climate change and were matched climate science with the general population on age and gender Poth et al. (2021) No privacy concerns A utilitarian approach Death rate considered applied because was taken as a fair measure the data sets were of COVID severity, publicly available conceptualization of analysis as an adaptive complex system

Explicability Detailed algorithmic explanation in supplementary material to the article

Yes, including a crossreference

Source: Authors.

applications illustrate the importance of big data’s velocity properties in natural language processing and the solution-finding (value) need to be completed within a few minutes for acceptable customer service (Kerravala, 2021). Table 11.3 summarizes which ethical principles for big data were given explicit consideration in the two studies from Table 11.2. The autonomy principle was excluded because it is more relevant to the autonomy of humans over machines than to empirical MMR studies (Herrmann, in press).

CONCLUSION The emergence and attention to the ethical considerations in the use of big data and associated technologies in MMR and the need for methodological transparency is central to this chapter. The implication for mixed methods researchers is twofold. First, mixed methods researchers will increasingly need to critically address the ethical issues and problems that are arising from the Fourth Industrial Revolution through its impact on economies, organizations, governments and societies in general. The use of big data analytics and associated technologies in decision-making processes, the application of algorithmic AI and associated ethical concerns (data biases, data fundamentalism and feminism, and privacy of the quantified

“self”) are all issues that need the critical eye of the social science researcher. Second, these digital applications and associated impacts across societies and economies will not only be topics of research, but also sources of data for future MMR. This has important implications. To start with, the processes involved in the development and training of AI models will need to be understood by the mixed methods researcher. Moreover, once MMR moves from batch analysis—the mode of analysis in its current use of big data (see the examples)—to real-time analysis to include big data’s velocity, MMR teams will need to be equipped with solid information and communication technology (ICT) skills. The extent to which such real time-analysis requires additional skilling in ethics versus batch processing is an interesting area for future research. MM researchers will need to build a knowledge base and critical eye in terms of the decisions, values and biases involved in the development and applications of these “intelligences” and the use of big data in MMR studies. In practical terms, the big data conceptual framework, definitions and ethical AI principles presented provide some foundational conceptual terms, ethical underpinnings and useful resources for MM researcher skilling. We anticipate that this will achieve greater levels of methodological transparency when documenting methodological choices and the ethical dimensions in an MMR study, in an increasingly complex data-driven world.

162

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

WHAT TO READ NEXT D’ignazio, C., & Klein, L. F. (2020). Data feminism. MIT Press.

This book provides seven principles and practical guidelines to counterbalance data bias. Herrmann, H., & Cameron, R. (forthcoming 2023). Responsible Mixed Methods Research (RMMR). In R. Cameron & X. Golenko (Eds.), Handbook of Mixed Methods Research in Business and Management. Edward Elgar.

This chapter describes a unified framework for responsible AI (RAI) governance, and synthesizes the ethical considerations for MMR and the RAI framework towards responsible MMR (RMMR). Mittelstadt, B. D., & Floridi, L. (Eds.). (2016). The ethics of biomedical big data (Vol. 29). Springer.

This pioneering book addresses the ethical challenges posed by biomedical big data and discusses key issues. Shamoo, A. E., & Resnik, D. B. (2009). Responsible conduct of research. Oxford University Press

This book provides an overview on ethics as an academic discipline. Internet-accessible summaries of the latter book with recent additions are available at: www.niehs.nih.gov/research/ resources/bioethics/whatis/index.cfm www.niehs. nih.gov/research/resources/bioethics/glossary/ index.cfm A free online course in AI Ethics is also available at: https://aiethicscourse.org/

REFERENCES Andreotta, M., Nugroho, R., Hurlstone, M., Boschetti, F., Farrell, S., Walker, I., & Paris, C. (2019). Analyzing social media data: A mixed-methods framework combining computational and qualitative text analysis. Behavior Research Methods, 51(4), 1766–1781. https://doi.org/10.3758/ s13428-019-01202-8 Arthur, K. N. A., & Owen, R. (2019). A micro-ethnographic study of big data-based innovation in the financial services sector. Journal of Business Ethics, 160(2), 363–375. https://doi.org/10.1007/ s10551-019-04203-x Ashta, A., & Herrmann, H. (2021). Artificial intelligence and fintech. Strategic Change, 30(3), 211– 222. https://doi.org/10.1002/jsc.2404 Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018).

The moral machine experiment. Nature, 563(7729), 59–64. https://doi.org/10.1038/s41586-0180637-6 Bell, E., & Wray-Bliss, E. (2010). Research ethics: Regulations and responsibilities. In D. B. Buchanan & A. Bryman (Eds.), The Sage handbook of organizational research methods (pp. 78–92). Sage. Berryhill, J., Heang, K. K., Clogher, R., & McBride, K. (2019). Hello, world: Artificial intelligence and its use in the public sector. OECD. https://oecd-opsi.o rg/wp-content/uploads/2019/11/AI-Report-Online. pdf Blackman, R. (2021). If your company uses AI, it needs an institutional review board. Harvard Business Publishing. Retrieved 14 April 2021 from: https://hbr.org/2021/04/if-your-company-uses-aiit-needs-an-institutional-review-board?utm_ m e d i u m = e m a i l & u t m _ s o u rc e = n e w s l e t t e r _ monthly&utm_campaign=strategy_activesubs& utm_content=signinnudge&deliveryName= DM127580 Borges, A. F. S., Laurindo, F. J. B., Spínola, M. M., Gonçalves, R. F., & Mattos, C. A. (2021). The strategic use of artificial intelligence in the digital era. International Journal of Information Management, 57, 102225. https://doi.org/10.1016/j.ijinfomgt. 2020.102225 Bornakke, T., & Due, B. L. (2018). Big–thick blending: A method for mixing analytical insights from big and thick data sources. Big Data & Society, 5(1), 1–16. https://doi.org/10.1177/205395171 8765026 Boyd, D. (2010). Privacy and publicity in the context of big data [keynote address]. The 19th International World Wide Web Conference, Raleigh, NC, USA. Bulut, O., & Poth C. (2022). Rapid assessment of communication consistency: Sentiment analysis of public health briefings during the COVID-19 pandemic. Public Health, 9(2), 293–306. https:// doi/10.3934/publichealth.2022020 Cain, L., MacDonald, A., Coker, J., Velasco, J. & West, G. (2019). Ethics and reflexivity in mixed methods research: An examination of current practices and a call for further discussion. International Journal of Multiple Research Approaches. 11(2), 144–155. https://doi.org/10.29034/ijmra. v11n2a2 Chang, V. (2015). Delivery and adoption of cloud computing services in contemporary organizations. IGI Global. https://doi.org/10.4018/9781-4666-8210-8 Collins, K. M. T., Onwuegbuzie, A. J., & Johnson, R. B. (2012). Securing a place at the table. American Behavioural Scientist, 56(6), 849–865. https://doi. org/10.1177/0002764211433799 Collins, K. M. T., Onwuegbuzie, A. J., Johnson, R. B., & Frels, R. K. (2013). Practice note: Using debriefing

ETHICAL ISSUES AND PRACTICES FOR MIXED METHODS RESEARCH

interviews to promote authenticity and transparency in mixed research. International Journal of Multiple Research Approaches, 7(2), 271–284. https://doi.org/10.5172/mra.2013.7.2.271 Crespo-Perez, G., & Ojeda-Castro, A. (2017). Convergence of cloud computing, internet of things, and machine learning. International Journal of Scientific and Technology Research, 6(07), 131–136. D’Ignazio, C., & Klein, L. F. (2020). Data feminism. MIT Press. Davenport, T. (2020). Beyond unicorns: Educating, classifying, and certifying business data scientists. Harvard Data Science Review 2(2). https://doi.org/ 10.1162/99608f92.55546b4a De Mauro, A., Greco, M., & Grimaldi, M. (2016). A formal definition of big data based on its essential features. Library Review, 65(3), 122–135. https:// doi.org/10.1108/LR-06-2015-0061 Dubois, E., & Ford, H. (2015). Trace interviews: An actor-centered approach. International Journal of Communication, 9, 2067–2091. European Union. (2020). The framework programme for research and innovation. https://op.europa.eu/ en/publication-detail/-/publication/3f5d8bd9d358-11eb-ac72-01aa75ed71a1 Feng, X., Jiang, Y., Yang, X., Du, M., & Li, X. (2019). Computer vision algorithms and hardware implementations: A survey. Integration, 69, 309–320. Fiaidhi, J., & Mohammed, S. (2019). Thick data: A new qualitative analytics for identifying customer insights. IT Professional, 21(3), 4–13. Fitsch, H., Jordan-Young, R., Kaiser Trujillo, A., Kraus, C., Roy, D., & Schmitz, S. (2020). Coalitionmaking and the practice of feminist STS in the time of COVID-19. Catalyst: Feminism, Theory, Technoscience, 6(2). https://doi.org/10.28968/cftt. v6i2.34640 Floridi, L., & Cowls, J. (2019). A unified framework of five principles for AI in society. Harvard Data Science Review, 1(1), 1–15. https://doi.org/10.1162/ 99608f92.8cd550d1 Floridi, L., Cowls, J., King, T. C., & Taddeo, M. (2020). How to design AI for social good. Science and Engineering Ethics, 26(3), 1771–1796. https://doi. org/10.1007/s11948-020-00213-5 Guszcza, J., Lee, M., Ammanath, B., & Kuder, D. (2021). Human values in the loop. www2.deloitte. com/us/en/insights/focus/cognitive-technologies/ design-principles-ethical-artificial-intelligence. html/#endnote-sup-11 Herrmann, H. (forthcoming 2023). Introducing the Systematic Science Mapping Framework: An innovative and mixed approach for macro scale reviews. In R. Cameron & X. Golenko (Eds.), Handbook of mixed methods research in business and management. Edward Elgar. Herrmann, H. (2022). The arcanum of artificial intelligence in enterprise applications: Toward a

163

unified framework. Journal of Engineering and Technology Management, 66(Oct.–Dec.) 101716. https://doi.org/10.1016/j.jengtecman.2022. 101716 Herrmann, H., & Cameron, R. (forthcoming 2023). Responsible Mixed Methods Research (RMMR). In R. Cameron & X. Golenko (Eds.), Handbook of Mixed Methods Research in Business and Management. Edward Elgar. Herrmann, H., & Masawi, B. (2022). Three and a half decades of artificial intelligence in banking, financial services, and insurance: A systematic evolutionary review. Strategic Change, 31(6), 549–569. https://doi.org/10.1002/jsc.2525 Herschel, R., & Miori, V. M. (2017). Ethics & big data. Technology in Society, 49, 31–36. https://doi. org/10.1016/j.techsoc.2017.03.003 Hesse-Biber, S. N., & Leavy, P. (2006). The practice of qualitative research. Sage. Hesse-Biber, S. (2010). Qualitative approaches to mixed methods practice. Qualitative Inquiry, 16(6), 455–468. doi:10.1177/10-77800410364611 Hesse-Biber, S., & Johnson, R. B. (2013). Coming at things differently: Future directions of possible engagement with mixed methods research. Journal of Mixed Methods Research, 7(2), 103–109. doi:10.1177/1558689813483987 Hesse-Biber, S., & Griffin, A. J. (2013). Internet-mediated technologies and mixed methods research. Journal of Mixed Methods Research, 7(1), 43–61. https://doi.org/10.1177/1558689812451791 Hillard, D., Purpura, S., & Wilkerson, J. (2008). Computer-assisted topic classification for mixedmethods social science research. Journal of Information Technology and Politics, 4(4), 31–46. https://doi.org/10.1080/19331680801975367 Holtrop, J. S., Potworowski, G., Green, L. A., & Fetters, M. (2019). Analysis of novel care management programs in primary care. Journal of Mixed Methods Research, 13(1), 85–112. https://doi. org/10.1177/1558689816668689 Hurtgen, H., Kerkhoff, S., Lubatschowski, J., & Möller, M. (2020). Rethinking AI talent strategy as automated machine learning comes of age. McKinsey. www.mckinsey.com/business-functions/ mckinsey-analytics/our-insights/rethinking-aitalent-strategy-as-automated-machine-learningcomes-of-age Isoaho, K., Gritsenko, D., & Mäkelä, E. (2021). Topic modeling and text analysis for qualitative policy research. Policy Studies Journal, 49(1), 300–324. https://doi.org/10.1111/psj.12343 Jaume-Palasí, L., Matzat, L., Spielkamp, M., & Zweig, K. A. (2021). AI ethics guidelines global inventory. AlgorithmWatch. https://algorithmwatch.org/en/ ai-ethics-guidelines-global-inventory/ Kerravala, Z. (2021) Why conversational AI is now ready for prime time. ZDNet. www.zdnet.com/

164

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

article/why-conversational-ai-is-now-ready-forprime-time/?ftag=TRE6a12a91&bhid=%7B%24e xter n al_id%7D& m id=% 7B% 24M E S S AGE _ ID%7D&cid=%7B%24contact_ id%7D&eh=%7B%24CF_emailHash%7D Leicht-Deobald, U., Busch, T., Schank, C., Weibel, A., Schafheitle, S., Wildhaber, I., & Kasper, G. (2019). The challenges of algorithm-based HR decisionmaking for personal integrity. Journal of Business Ethics, 160(2), 377–392. https://doi.org/10.1007/ s10551-019-04204-w Leijnen, S., & Veen, F. V. (2020). The neural network zoo. Proceedings, 47(1), 9. https://doi. org/10.3390/proceedings47010009 Liao, Y., Loures, E. R., Deschamps, F., Brezinski, G., & Venâncio, A. (2018). The impact of the fourth industrial revolution. Production, 28(0), 1–18. https://doi.org/10.1590/0103-6513.20180061 Loechner, J. (2016). 90% Of Today’s Data Created In Two Years. MediaPost. www.mediapost.com/ publications/article/291358/90-of-todays-datacreated-in-two-years.html Martin, K. (2019). Ethical implications and accountability of algorithms. Journal of Business Ethics, 160(4), 835–850. https://doi.org/10.1007/s10551018-3921-3 Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & Delgado López-Cózar, E. (2018). Google scholar, web of science, and scopus: A systematic comparison of citations in 252 subject categories, Journal of Informetrics, 12(4), 1160–1177. https://doi. org/10.1016/j.joi.2018.09.002 Mertens, D. M., Bazeley, P., Bowleg, L., Fielding, N., Maxwell, J., Molina-Azorin, J. F., & Niglas, K. (2016). Expanding thinking through a kaleidoscopic look into the future: Implications of the Mixed Methods International Research Association’s Task Force report on the future of mixed methods. Journal of Mixed Methods Research, 10(3), 221–227. https:// doi.org/10.1177/1558689816649719 Mittelstadt, B., & Floridi, L. (2016). The ethics of biomedical big data. Springer. Moats, D., & Borra, E. (2018). Quali-quantitative methods beyond networks: Studying information diffusion on twitter with the modulation sequencer. Big Data & Society, 5(1), 205395171877213. https://doi.org/10.1177/2053951718772137 Moyer, E. (2021). Facebook privacy lawsuit over facial recognition leads to $650M settlement. Cnet. www.cnet.com/news/facebook-privacy-lawsuitover-facial-recognition-leads-to-650m-settlement/ Müller, J., Fàbregues, S., Guenther, E. A., & Romano, M. J. (2019). Using sensors in organizational research-clarifying rationales and validation challenges for mixed methods. Frontiers in Psychology, 10(May), 1188. https://doi.org/10.3389/fpsyg. 2019.01188 Müller, V. C., & Bostrom, N. (2016). Future progress in artificial intelligence: A survey of expert opinion.

In Fundamental issues of artificial intelligence (pp. 555–572). Springer. Nadkarni, P. M., Ohno-Machado, L., & Chapman, W. W. (2011). Natural language processing. Journal of the American Medical Informatics Association, 18(5), 544–551. https://doi.org/10.1136/amiajnl2011-000464 Nelson, H. J. (2021). In J. Liebowitz (Ed.), Data analytics & AI (pp. ix–xv). Taylor & Francis. Nicholls, S., Langan, S., & Benchimol, E. (2016). Reporting and transparency in Big Data. In B. Mittelstadt & L. Floridi (Eds.), The ethics of biomedical big data (pp. 95–115). Springer. https://doi. org/10.1007/978-3-319-33525-4 Nunez-Mir, G. C., Iannone, B. V., Pijanowski, B. C., Kong, N., & Fei, S. (2016). Automated content analysis. Methods in Ecology and Evolution, 7(11), 1262–1272. https://doi.org/10.1111/2041210x.12602 O’Halloran, K. L., Tan, S., Pham, D.-S., Bateman, J., & Vande Moere, A. (2016). A digital mixed methods research design: Integrating multimodal analysis with data mining and information visualization for Big data analytics. Journal of Mixed Methods Research, 12(1), 11–30. https://doi.org/10.1177/ 1558689816651015 O’Halloran, K. L., Tan, S., Wignell, P., Bateman, J. A., Pham, D.-S., Grossman, M., & Moere, A. V. (2019). Interpreting text and image relations in violent extremist discourse: A mixed methods approach for big data analytics. Terrorism and Political Violence, 31(3), 454–474. O’Mahony, N., Campbell, S., Carvalho, A., Harapanahalli, S., Hernandez, G. V., Krpalkova, L., Riordan, D., & Walsh, J. (2020). Deep learning vs. traditional computer vision. In Arai, K. & Kapoor, S. (Eds.), Advances in Computer Vision (pp. 128– 144). Springer International. https://doi.org/ 10.1007/978-3-030-17795-9_10 Onwuegbuzie, A. J., & Corrigan, J. A. (2014). Improving the quality of mixed research reports in the field of human resource development and beyond. Human Resource Development Quarterly, 25(3), 273–299. doi:10.1002/hrdq.21197 Poth, C. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Poth, C. N., Bulut, O., Aquilina, A. M., & Otto, S. J. (2021). Using data mining for rapid complex case study descriptions: Example of public health briefings during the onset of the COVID-19 pandemic. Journal of Mixed Methods Research, 15(3), 348–373. Preissle, J., Glover-Kudon, R., Rohan, E. A., Boehm, J. E., & DeGroff, A. (2015). Putting ethics on the mixed methods map. In Hesse-Biber, S. N. & Johnson, R. B. (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry. Oxford University Press.

ETHICAL ISSUES AND PRACTICES FOR MIXED METHODS RESEARCH

Ratia, M., Myllärniemi, J., & Helander, N. (2019). The potential beyond IC 4.0: the evolution of business intelligence towards advanced business analytics. Measuring Business Excellence, 23(4), 396-410. https://doi.org/10.1108/mbe-12-2018-0103 Ruiz-Real, J. L., Uribe-Toril, J., Torres, J. A., & De Pablo, J. (2020). Artificial intelligence in business and economics research. Journal of Business Economics and Management, 22(1), 98–117. https:// doi.org/10.3846/jbem.2020.13641 Russell, S. J., & Norvig, P. (2020). Artificial intelligence: A modern approach (4th ed.). Prentice Hall. Sangeetha, S., & Sreeja, A. K. (2015). No science no humans, no new technologies no changes: “Big data a great revolution”. International Journal of Computer Science and Information Technology, 6(4), 3269–3274. Satyanarayanan, M., Bahl, P., Caceres, R., & Davies, N. (2009). The case for VM-based cloudlets in mobile computing. IEEE Pervasive Computing, 8(4), 14–23. https://doi.org/10.1109/mprv.2009.82 Schlenker, L., & Minhaj, M. (2020). Machine intelligence and managerial decision-making. In Liebowitz, J. (Ed.), Data analytics & AI. Taylor & Francis. Sena, V., Bhaumik, S., Sengupta, A., & Demirbag, M. (2019). Big data and performance. British Journal of Management, 30(2), 219–228. https://doi. org/10.1111/1467-8551.12362 Serrao, L. G. (2020). Grokking machine learning. Manning. Shamoo, A., & Resnik, D. (2009). Responsible conduct of research. Oxford University Press. Sivarajah, U., Kamal, M. M., Irani, Z., & Weerakkody, V. (2017). Critical analysis of big data challenges

165

and analytical methods. Journal of Business Research, 70, 263–286. https://doi.org/10.1016/j. jbusres.2016.08.001 Smets, A., & Lievens, B. (2018, October). Human Sensemaking in the Smart City: A Research Approach Merging Big and Thick Data. In Wieland, J. (Ed.), Ethnographic Praxis in Industry Conference Proceedings, 1, 179–194. Stadnick, N. A., Poth, C. N., Guetterman, T. C., & Gallo, J. J. (2021). Advancing discussion of ethics in mixed methods health services research. BMC Health Services Research, 21(1), 577–586. https:// doi.org/10.1186/s12913-021-06583-1 Thomas, S. L., Nafus, D., & Sherman, J. (2018). Algorithms as fetish. Big Data & Society, 5(1), 205395171775155. https://doi.org/10.1177/ 2053951717751552 Thompson, C. J. (2019). The ‘big data’ myth and the pitfalls of ‘thick data’ opportunism. Journal of Marketing Management, 35(3–4), 207–230. Wiguna, T., Wigantara, N. A., Ismail, R. I., Kaligis, F., Minayati, K., Bahana, R., & Dirgantoro, B. (2020). A four-step method for the development of an ADHD-VR digital game diagnostic tool prototype for children using a DL model. Frontiers in Psychiatry, 11, Article 829. https://doi.org/10.3389/ fpsyt.2020.00829 Yampolskiy, R. V. (2013). Artificial intelligence safety engineering. In Müller, V. (Ed.), Philosophy and theory of artificial intelligence (pp. 389–396). Springer. Zeroual, I., & Lakhouaja, A. (2018). Data science in light of natural language processing. Procedia Computer Science, 127, 82–91.

12 Building the Logic for an Integrated Methodology: Mixed Method Grounded Theory as an Example of Constructing a Methodology to Guide Design and Integration E l i z a b e t h G . C r e a m e r, C a s s a n d r a McCall and Cherie D. Edwards

Mixed method research is frequently paired with other research methods (Creamer & Schoonenboom, 2018b). Attesting to this capacity for adaptability to diverse paradigms and approaches, several methodologically oriented authors have participated in the conversation about the ways that mixed methods can be partnered with grounded theory (e.g., Creamer, 2021; Guetterman et  al., 2017; Howell Smith et  al., 2020; Johnson et  al., 2010; Johnson & Walsh, 2019). When research methods are approached exclusively as a set of procedures, this type of pairing is not always accomplished with the reflexivity necessary to develop a coherent rationale to guide design choices, including integration. This kind of reflexivity requires sufficient familiarity with the literature about the research methods that are paired to be able to recognize shared philosophical assumptions, consider the unique contribution of each, the potential for synergies in the pairings, tensions that are introduced and possible ways that the research design might unfold. Mixing or integration of qualitative and quantitative approaches is at the centre of the way that mixed method research is defined. This is its major distinction from multimethods. An integrated methodology is the product of the meaningful

integration of two or more research approaches that have complementary assumptions. Some experts in the field define mixed methods in a way that is compatible with inter-method mixing by pointing out that mixing can occur at many levels, including data, methods or perspectives (theory). Some frame their definition of mixed method research in an expansive way that is compatible with an integrated methodology that adopts a dialectical paradigm. For example, Johnson and Onwuegbuzie (2007) define mixed methods not just simply as mixing qualitative and quantitative data, or data collection procedures, but as “an approach to knowledge (theory and practice) that attempts to consider multiple viewpoints, perspectives, positions, and standpoints” (p. 113).

Purpose This chapter grapples with the task of constructing a coherent rationale for the partnering of mixed methods with a qualitative approach like grounded theory. It introduces a theoretical approach to integration. With mixed method grounded theory (MM-GT) as an example, the chapter proposes a strategy to embed multilevel mixing in the way

BUILDING THE LOGIC FOR AN INTEGRATED METHODOLOGY

that an integrated methodology is conceived by offering a template to extract information about the compatibilities and tensions between the philosophical assumptions and core procedures of two or more research methods. Four interlinked visual displays are designed to help make it possible to visualize what an integrated methodology looks like in practice. A series of interconnected tables and figures carry forward elements introduced in an initial figure (Figure 12.1) that identifies multiple ways that integration can occur when compatible research methods are partnered in a research project. We introduce examples from the literature to advance the discussion of the potential of MM-GT to function as an integrated methodology by considering some of the ways that it is embodied in two different approaches to research design. The chapter draws to a close with a section that explores the tensions that can be introduced when mixed methods and grounded theory are paired in a thoughtful and informed way. This chapter builds on previous work by the authors, including how to approach mixed method research in a fully integrated way (Creamer, 2018a; 2022a) that extends to embedding a dialectical logic in the use of visual displays to advance analytical insight (Creamer & Edwards, 2019; Reeping & Edwards, 2020). At the same time, it contributes to critical conversations of mixed methods and

167

grounded theory compatibility by drawing from the authors’ prior work in examining philosophical and paradigmatic comparisons across grounded theory traditions (McCall & Edwards, 2021) and grounded theory implementation and theory development (McCall et al., 2021). The first author has elaborated a methodological rationale for the pairing of a dialectical approach to mixed methods with a constructivist approach to grounded theory in greater detail elsewhere (Creamer, 2021; Creamer, 2022b). For a further discussion of best practices for design and implementation of MM-GT; see also Chapter 21 (this volume).

MIXED METHOD GROUNDED THEORY AS A METHODOLOGY When developed in a way that is informed by knowledge of contemporary methodological literature and very likely prolonged debates among team members, MM-GT offers a fertile example to illustrate how the logic for an integrated methodology can be constructed and the ways this construction is carried forward to elements of the research design. MM-GT is a research methodology that embeds a dialectical logic in the constant

Paradigm Conceptu al Framewo rk Research

Methods

Research

Design

Data Analytic P

rocedures

Reporting

Figure 12.1  The figure conceptualizes an integrated methodology as an example of intentional mixing at many levels. It is only a partial list of the ways that integration could occur when another methodology is paired with mixed methods.

168

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

comparative method and grounded theory procedures to develop a context-contingent theoretical framework or to elaborate an existing one (Creamer, 2021). The pairing of mixed method with grounded theory has been used with a variety of research designs and approaches (Creamer, 2021; Johnson & Walsh, 2019; Walsh, 2015). It can be approached in ways that create an integrated enquiry logic where the differences between the methods are not erased but leveraged to achieve a whole that is greater than the sum of its parts. This type of partnership has been variously referred to as methodological triangulation (Denzin, 1978), theoretical triangulation (Kushner & Morrow, 2003), inter-method mixing (Creamer & Schoonenboom, 2018), and methods braiding (Watson, 2019).

VISUALIZING AN INTEGRATED METHODOLOGY AS A COHERENT APPROACH TO MULTILEVEL MIXING A key element of the design of a mixed method study is the decision about what levels to mix relative to its aims, purposes, philosophical and sociopolitical commitments (Greene, 2015). Greene (2015) broadens views about mixing by characterizing multilevel mixing as a key characteristic of the design of a mixed method study, writing, “mixing at multiple levels [emphasis hers] is integral to the characteristics of mixed method approaches” (p. 607). She suggests that multilevel mixing can occur not just at the level of methods, but also through mixing methodologies, philosophies, academic discipline or theory. Mixing at multiple levels is a way to meaningfully engage differences, Greene (2015) observes. Fetters and Molina-Azorin (2017) suggest that the term “integration” rather than “mixing” is appropriate when the process of integration is conceived in a fully integrated way. Endorsing the idea of a fully integrated approach to integration, Fetters and Molina-Azorin write: “We wish to emphasize that a key to advancing conceptualizations of ‘integration’ in the context of mixed method research requires consideration during all phases of integration” (2019, p. 13). Fetters and Molina-Azorin provide a comprehensive list of 15 possible ways that integration might occur (2017, Table 1, p. 294) that range from philosophical or theoretical, to team dynamics, the literature and elements of the methods, and dissemination and reporting. The priority they place on multilevel integration can be applied to constructing a

methodological framework to guide integration when complementary methods are paired. Adapting the language from Greene (2015) and her use of the term “mixing” rather than “integration”, Figure 12.1 uses a two-dimensional pyramid to visualize different ways that multilevel mixing might occur in an integrated methodology. The layers shown in the pyramid match Greene’s language about mixing at different levels from the most abstract at the top (i.e., paradigm, theory) to the more technical toward the bottom. For another example of visualizing the interactions between methods in mixed methods, see also Chapter 8 (this volume). One of the challenges of constructing the logic for pairing research methods is to employ them in a way that honours the integrity of the foundational assumptions of the enquiry logic for each research method. This more deliberative approach differs from one where some elements of a research method are adopted without recognition of the wider enquiry logic. This is the case, for example, in the very common practice of assuming that grounded theory is simply an approach to coding data, without considering that it is embedded in a wider logic designed to build theory inductively. The same kind of “method slurring” (Baker et  al., 1992, p. 1355) occurs in research that adapts the mixed method label when there is no plan to integrate data or analytical procedures in intentional ways. The next section of the chapter explores the idea of MM-GT as an integrated methodology that is scaffolded by complementarities between the core philosophical assumptions of each research method. It takes the position that methodological reflexivity can be extended by thoughtful engagement with the literature and by identifying the potential synergies that can emerge from the pairing of mixed methods and grounded theory.

CONSTRUCTING THE SCAFFOLDING FOR AN INTEGRATED METHODOLOGY A methodology is a theory of how the research process should unfold that binds method (procedures, techniques) with philosophy (Schwandt, 2007). A methodology supplies an overriding logic that links how parts of a research study are connected or unfold (Hesse-Biber, 2018). An integrated methodology is the product of an informed partnering of two or more research approaches that are consonant in that they share compatible philosophical and methodological assumptions, while at the same time each making a unique contribution.

BUILDING THE LOGIC FOR AN INTEGRATED METHODOLOGY

The pairing of mixed method with grounded theory has been used with a variety of research designs and approaches (Creamer, 2021; Johnson & Walsh, 2019; Walsh, 2015). Referring to it as “mixed grounded theory” (MGT), Johnson and Walsh observe: “There are perhaps an infinite number of versions of MGT to be explored and ‘invented’ by practicing researchers in the production of knowledge” (2019, p. 523). Reflecting on the experience of completing a complex, multiphase research project in information systems using MM-GT, Isabelle Walsh concluded that every project is like a specialized ecosystem that can be considered its own unique methodology that combines beliefs and practices with a specific combination of methods and techniques (Walsh, 2015). The variety of approaches to grounded theory is a strength, not a weakness, of the methodology (Hadley, 2019). Part of the dynamism in the pairing of mixed methods with grounded theory is its adaptability to meet the purpose and goals of a project and to adapt to dynamics in the research environment. Another is its adaptability to diverse paradigmatic/philosophical orientations, including dialectical pluralism (Creamer, 2021) and pragmatism (Morgan, 2020). One of the challenges in its pairing with mixed methods is to navigate between the distinct schools of thought that have emerged over the years about grounded theory (Guetterman et  al., 2017; Howell Smith et al., 2020). Table 12.1 offers a template to build a rationale for an integrated methodology by extracting information about the compatibilities and tensions between the philosophical assumptions and core procedures of two or more research methods. Table 12.1 is organized by the levels of mixing depicted in Figure 12.1. Using MM-GT undertaken with a dialectical paradigm as an example, this exercise embeds multilevel mixing in the way an integrated methodology is conceived. The shading in the table distinguishes between philosophical dimensions (shaded) and those related to research methods or procedures (not shaded). The table is organized horizontally with a column that highlights the unique contribution of grounded theory and one that recognizes the contribution of mixed methods. A blank cell indicates that this is not a topic where the method makes a contribution. The third column identifies some of the synergies that can emerge through the pairing. This could be, for example, the advantage of being able to adapt the procedures for use with a secondary database. Additional columns would be needed if more than two research methods were involved. Table 12.1 offers a framework to build a coherent methodological framework to inform design decisions. Ideally, it would be constructed during

169

the planning stage after the research purpose was established, but before finalizing the research procedures. The table is conceived as creating an integrated methodology that pairs another research method with mixed methods, but nothing would prohibit its use with research that is informed by more than one qualitative method. It is possible to move from the realm of the abstract to one that is more practical by considering the implications of the data points in Table 12.1 to what is prioritized in the design of a MM-GT study. Some of these are linked to grounded theory, while others emerge synergistically in the MM-GT pairing. Design implications that tie most closely to the foundational assumptions of grounded theory include: (1) an emphasis on theory development and refinement; (2) focus on a social process with a temporal dimension; and (3) a protocol for coding data inductively. Other design features that emerge synergistically in the pairing of grounded theory with mixed methods include: (1) a three-prong enquiry logic that includes inductive, abductive, and deductive reasoning as an emergent quality; (2) an element of verification that can occur as a separate phase of the research process or that can be embedded in the analysis through an abductive process; and (3) dissonance and paradox as an element of complexity.

DISTINGUISHING TWO APPROACHES TO MM-GT RESEARCH DESIGNS The wide variety of ways that MM-GT has been implemented in practice makes it challenging to isolate design features using conventional means that can be helpful to others setting out to pursue this kind of study in a way that thoughtfully and intentionally integrates the two research traditions. Meaningful approaches to integration introduce complexity and an emergent quality that make it unlikely that design features can be cemented in place during the planning phase. In reflecting about MM-GT, Walsh (2015, p. 20) advises that “the decision to mix data and method, and when to do so, are not set at the start of the research project but emerge as the research proceeds”. Johnson and Walsh (2019) concentrated on the way the constant comparative method was applied in six illustrative cases of MM-GT. Grounded theorists define constant comparative analysis as the process of continuously comparing new data with data collected and analyzed previously (Walsh, 2015). Johnson and Walsh (2019) observed that

170

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 12.1  A multilevel framework to identify the unique contribution of each method and the potential synergies of MM-GTM, organized by philosophical assumptions and research procedures Level of mixing

Contribution of GT

Contribution of MMR

Paradigm compatibility

Adaptable to diverse paradigms.

Conceptual or explanatory framework

Focus on social processes with a temporal dimension.

Core assumptions of the research method

Purpose to generate an explanatory theory inductively. A protocol for coding data inductively.

Attention to integration.

Research design

Ensures there is a strong exploratory component.

Adding verification.

Data

Endorsement of the idea that all is data.

Incentive to utilize diverse sources of data.

Analytical procedures

Theoretical coding and sampling.

Embraces the use of joint displays for integration.

Reporting

Visualizing theoretical models generated inductively.

Visual models that incorporate qualitative and quantitative data.

Potential synergies from integrating MMR and GT Engaging more than one paradigm. The potential not only to develop theory but to recognize the existing literature and to elaborate or refute an existing theory. An abductive approach to theoretical sampling that generates tentative hypotheses and simultaneously moves between exploration and verification. Expanded range of research purposes. Adaptable to complexity and diverse approaches to design. Potential to apply the constant comparative method to data that has been collected already. Ability to use a coding scheme that has both inductively and deductively derived codes. Reflexivity about team dynamics in reporting.

Note: The shading in the left column distinguishes between the philosophical dimensions (shaded) and those related to research method (not shaded). Source: Author created.

among the six cases in MM-GT the constant comparative method was equally as likely to be confined to the qualitative phases as it is to an integrated approach, where it is applied to both qualitative and quantitative data. Two of the six cases they identified are fully integrated in that they applied the constant comparative method to both the qualitative and quantitative data throughout the analytical process. The two examples are bookends on a continuum of mixing across levels depicted in Figure 12.1. The authors of the two examples each integrated mixed methods with grounded theory in

different ways. The first embodies mixing during analysis. The second demonstrates how a team can design a context-sensitive approach that is an integrated methodology in that mixing is accomplished at multiple levels. From the discipline of management, the first example by Kaplan and Duchon (1988) is more conventional. The pair of collaborators adapted a core mixed method design and with one exception compartmentalizing the qualitative and quantitative analytical procedure in a concurrent or parallel design. Grounded theory procedures were confined to the qualitative phase. The second example by Westhues et  al.

BUILDING THE LOGIC FOR AN INTEGRATED METHODOLOGY

(2008) embodies multilevel mixing pictured in Figure 12.1. This multi-year project teamed mixed methods, grounded theory, and participatory action research. It is mixing during analysis in that the constant comparative method was applied to both qualitative and quantitative data. This was achieved by treating the entire data set as one. Both examples approached the analytical process differently. Each extended ideas about how to pair mixed methods with grounded theory by illustrating that data analysis and data collection do not necessarily have to co-occur. The constant comparative method was activated after all data were collected in both examples. An additional contribution is to centre the role of an abductive logic in the analytical process. For Kaplan and Duchon (1988), the abductive process of generating multiple possible explanations for unexpected findings was added to reconcile contradictions between the results emerging from the separate analysis of the qualitative and quantitative data. Procedural diagrams for each of the examples differ. One of the conventional templates that depict the timing of data collected and analysis can be adapted for the first example. The procedural diagram presented in the second example narrows the focus to analysis and integration. Each example is discussed in greater detail below.

Example 1: Maintaining the Distinction Between the Qualitative and Quantitative Strands The first example by Kaplan and Duchon (1988) was accomplished by a pair of collaborators with

171

distinct methodological expertise. Firm differences in expertise about research methods kept the qualitative and quantitative strands separate. The pair used an exploratory, two-phase sequential mixed methods design where multiple forms of qualitative data were collected first, followed by the construction of a questionnaire and analysis of data it produced. An unanticipated iterative phase was introduced in the analysis when contradictions between the inferences drawn from the independent analysis of the qualitative and quantitative data, and the construction of a grounded theory model at the mid-point, meant that the researchers approached verification by reanalyzing the quantitative data. Kaplan and Duchon (1988) did not include a procedural diagram in their article to demonstrate the flow of the data collection and analysis. Table 12.2 is a generic visualization of how the analytical procedures of MM-GT might unfold when grounded theory procedures are confined to the qualitative strand. Mixing can be categorized using conventional terminology such as merging (Guetterman et al., 2021). The research moved from a separate analysis of the qualitative and quantitative data to an integrative exchange of both sources of data, and then a return to the separate analysis of the two sources of data guided by themes or hypotheses that were generated during the second phase. Table 12.2 imagines the last phase of analysis as returning to an integrative mindset in the construction of a final grounded theory model. Table 12.2 depicts an analytical process where the researcher(s) tackled, as Kaplan and Duchon (1988) did, contradictory findings by adding a step in the analytical process and a subsequent phase involving verification.

Table 12.2  Visualizing the analysis in a generic MM-GTM design with an abductive component and an iterative loop Phase

Qualitative phase

Phase 1

Generate a theoretical model inductively from qualitative data.

Phase 2

Phase 3 Phase 4

Mixing phase

Quantitative phase Analyze quantitative survey data informed by the literature.

Identify tensions between initial findings; introduce an abductive logic to generate new themes or hypotheses to pursue. Revisit qualitative data to refine and verify new themes. Construct a visualization of an integrated theoretical model.

Source: Author created.

Revisit quantitative data to refine and verify new themes.

172

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Example 2: An Embedded Approach to the Constant Comparative Method Westhues et  al. (2008) provide a multifaceted example of research design with a three-way pairing: mixed methods, grounded theory and community-based participatory action research. This team of researchers set out with the purpose to develop a conceptual framework to guide practice in community mental health organizations in Canada. The authors use MM-GT modeling to illustrate the intersections between culturally sensitive procedures and community mental health organizations. The study models reflexivity about constructing an integrated methodology that builds a conceptually and methodologically cohesive study by integrating at the multiple levels depicted

in Figure 12.1. Table 12.3 extends the shell of multiple possible levels of mixing introduced in Figure 12.1 and applied in Table 12.1 by using it to map the levels of mixing evident in Westhues et al. (2008). It maintains the shading that distinguishes components related to the philosophical dimensions from those related to design, research methods, and reporting. An added strategy for integration is that the authors leverage the use of visual displays to illustrate connecting factors and concepts for each stage of the research process. The pivotal role of the philosophical dimension and sociopolitical commitments is evident in the team’s commitment to be “multi-perspectival” (pp. 701, 703). The philosophical dimension and a commitment to a dialectical paradigm provide coherence that spans the research methods.

Table 12.3  Deconstructing the levels of mixing in Westhues et al. (2008) with the multilevel framework Level of mixing

Synergies from integrating MMR and GT from Table 12.1

Different levels of mixing in Westhues et al. (2008)

Paradigm compatibility

Adaptable to diverse paradigms.

Conceptual or explanatory framework

The potential not only to develop theory but to recognize the existing literature and to elaborate or refute an existing theory. An abductive approach to theoretical sampling that generates tentative hypotheses and simultaneously moves between exploration and verification. Expanded range of research purposes.

The authors characterize their research as being intentional about adapting a multi-perspectival stance that includes team members and stakeholders. Table 12.1 depicts an initial conceptual framework from the literature that served as a sensitizing concept to guide data collection and analysis. As depicted in Figure 12.2, the constant comparative method was applied to all the data.

Core assumptions of the research method

Research design

Data

Analytical procedures

Reporting

The purpose was to integrate findings from four sub-projects into a conceptual framework to guide practice. It includes a phase for verification. Incentive to utilize diverse sources of data. The authors used multiple sources of data, including interviews with key informants, a web survey, focus groups, and case studies. Developing a coding scheme that has both Depicted in Figure 12.2 showing all inductively and deductively derived the data sources and a three-stage codes. analytical process that moved from inductive to abductive, then to a quantitative verification stage. Reflexivity about team dynamics and Depicted in Figure 12.1: Taking Culture community partnerships. Seriously in Community Mental Health Committee Structure.

Note: The shading in the left column distinguishes between the philosophical dimensions (shaded) and those related to research method (not shaded). Source: Author created.

BUILDING THE LOGIC FOR AN INTEGRATED METHODOLOGY

Observing that a dialectical perspective strengthened their analysis, Westhues et  al. note: “The interactive and dialectic use of mixed data helped discover the paradoxes and contradictions in what we had been told, as well as the similarities and agreements” (p. 714). Westhues et al. (2008) provide a figure that represents how the constant comparative method was embedded across three levels of analysis and four sources of data (Figure 2, p. 704). The figure identifies the first level of analysis as inductive; the second as abductive; and the third as prioritizing verification. The express inclusion of abduction as a second level of analysis supports Morgan’s (2020, p. 68) assertion that, at its core, grounded theory procedures are abductive in that inferences or tentative hypotheses are developed to explain observations made from the data. Westhues et al.’s figure recognizes that emerging findings were considered in the light of the literature. Figure 12.2 reproduces (with copyright permission) the procedural diagram representing steps in the analysis from Westhues et  al. (2008). It provides an example of how the constant comparative method can be applied to both qualitative and quantitative data throughout the analytical process with a single data set that merges qualitative and quantitative data. The procedural diagram

173

provided by these authors depicts a feedback loop but neglects the team dynamics so central to other visuals in their report. Figure 12.3 (below) presents another way to visualize the multilevel logic of integration evident in the Westhues et al. (2008) figure. It adds nuance to Figure 12.2 by recognizing all three research methods and by drawing more attention to the philosophical and sociopolitical commitments that are particularly central to this research project, and the ways that the participatory action research model influenced their collaborative approach to teamwork and interaction with community members from multiple linguistic and cultural groups. In Figure 12.3, the objectives of the Westhues et al. (2008) project to identify culturally competent practices to guide practice in a communityfocused mental health organization and to leverage multiple data strands to develop a conceptual framework are on the outside of the figure to illustrate their overriding role in informing the entire study. At the centre of the figure, we placed the methodological frameworks used to conduct the research (i.e., community-based research, grounded theory). Because each of these methodologies carries its own philosophical underpinnings and is executed through multiple methods,

Figure 12.2  Taking culture seriously in community mental health data analysis progression

174

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 12.3  Conceptualizing community-focused mental health to conduct a cohesive exploration, these methodologies are integrated. Furthermore, with each of the objectives tied to one or both of these methodological frameworks, the research objectives become another stage in which integration occurs. This conceptualization of the project acknowledges that some levels of mixing are likely to play a larger role in the research design than others.

The Role of Verification As an extension of the differences in their approaches to the constant comparative method, the two examples differ in their approaches to verification and whether it is embedded in analytical process or treated as a separate step in the analytical process. With its two-phase sequential design, Kaplan’s and Duchon’s (1988) approach is to position verification as the penultimate step of the analytical process. This follows the model put forward by Johnson et al. (2010) where quantitative analytical procedures are used to verify inferences drawn from the grounded theory developed qualitatively. Verification is treated differently in the example by Westhues et al. (2008) in that it is embedded in two of three analysis phases depicted in their flowchart (Figure 12.3). This approach is consistent with a definition of abduction that positions it as an approach to analysis that moves back and forth between an exploratory and confirmatory stance using the constant comparative method (Suddaby, 2006). Verification, along with attitudes about the role of literature and the weight given to recognizing

divergent views as a cluster, provide a bellwether of the differences between the schools of thought in grounded theory. The originators of grounded theory and some other leaders in the field, including Charmaz (2014), resist the idea that there is a need to verify findings or to explore exceptions (Dey, 1999).

UNEASY ALLIANCES: THE PAIRING OF MIXED METHODS WITH CLASSIC GROUNDED THEORY Although there are always differences in the way that research methodologists position a research method and define terms, the now more than 50 years of the history of grounded theory introduce some tensions when it is paired with mixed methods. Foremost among these is navigating different points of view about core grounded theory procedures (Apramian et al., 2017; Rieger, 2018; Sebastian, 2019). There are lively differences in viewpoints among members of this community on a variety of topics, including the role of the literature, acknowledgement of exceptions and openness to recognizing verification as part of the procedures. Hadley suggests that there is fluidity between the viewpoints, writing: “The variety of approaches to GTM need not be constructed as contradictory, but as part of an interactive, interdependent network” (Hadley, 2019, p. 568). Classic grounded theory was developed using a positivist paradigm (Lillemor & Hallberg,

BUILDING THE LOGIC FOR AN INTEGRATED METHODOLOGY

2006). This paradigm positions reality as an external, unyielding truth that can be explored through observation (McCall & Edwards, 2021). This underpinning foundation to the methodology has significantly influenced the ways that classic grounded theorists engage in their work at all phases and stages of the research process, from project ideation to formulating theory. At the outset of a study, Glaser (Glaser & Strauss, 1967) encourages researchers to “literally ignore the literature” (p. 37) and established frameworks for the topic under study. Moreover, Glaser contends that this limited understanding of a topic allows researchers to minimize preconceptions and remain open to a more robust inductive process of problem identification and exploration. This tension conflicts with nearly every other research methodology that requires and relies on an indepth literature review for problem identification and problem scoping (Dunne, 2011; McCall & Edwards, 2021). Similar tensions exist among the overall methodological process of classic grounded theory, including data collection and analysis, further underscoring the tensions that are introduced when it is paired with other research paradigms and approaches. While classic grounded theory relies on a constant comparative analysis, as carried through Straussian (i.e., pragmatic) and constructivist grounded theory (Charmaz, 2006, 2014), this process heavily relies on the iterative and inductive analysis of researcher observations and field notes (Glaser & Strauss, 1967). This process has been criticized by some as being vague and esoteric (Clarke, 2007) due to its lack of procedural description and heavy reliance on a researcher’s innate inclinations to move a study forward. Because of the positivist influence of classic grounded theory and the insistence that researchers avoid the literature in framing a study, this process simultaneously emphasizes the role of the researcher as observer, yet absolves them of interpretive responsibility, which significantly conflicts with methodologies established using pragmatic, constructivist, and feminist paradigms. Due to the tensions that have just been reviewed, the potential for the type of synergies depicted in Table 12.1 are limited when a mixed method approach is paired with a classic or positivist approach to grounded theory. This pairing would interfere with the ability to achieve the type of multilevel mixing identified in Figure 12.1 and Table 12.1. It would offset the potential for at least three of the synergies identified in Table 12.1: the potential (1) to recognize the existing literature and theoretical models evident in the literature; (2) to develop a coding scheme that involves both inductively and deductively derived coding

175

categories; and (3) to apply an abductive approach to theoretical sampling that incorporates an element of verification. Adhering to the foundational assumptions of classic grounded theory is more compatible with a positivist approach to mixed methods that assigns different purposes to each strand, often theory generation and theory testing, and limits the interaction between them during data collection and analysis. This approach pairs methods without creating a distinct integrated methodology.

DISCUSSION The literature in mixed methods, particularly that emerged in the 1990s, has been criticized for emphasizing design elements and procedures at the expense of a theoretical framework (cf., Hesse-Biber, 2018). This chapter leverages an expansive definition of mixed methods that embraces it not only as mixing data collection and analytical procedures, but as having the potential to integrate methods, theoretical positions, and multiple perspectives. We set out with the practical goal of exploring how to construct a wellinformed rationale for the pairing of one or more qualitative methods with mixed methods in a way that provides a framework to guide a coherent approach to multilevel mixing. It elaborates a methodological rationale for pairing mixed methods and grounded theory to underscore the assertion, as made by Howell Smith et al. (2020), that the methods are complementary. The discussion in the final section alerts those using MM-GT that while compatible with most approaches to grounded theory, there are some irreconcilable tensions introduced when mixed methods is paired with a classic or formal approach to grounded theory. A fully integrated approach to mixed methods has been envisioned in several ways over time. A fully integrated approach to mixed methods is generally framed as one where integration is embedded in procedures throughout the research process (e.g., Creamer, 2018a; Tashakkori & Teddlie, 2003; Teddlie & Tashakkori, 2009). The commentary provided here adds to this discussion by demonstrating a theoretical way to conceptualize a fully integrated approach to mixed methods. Offering Westhues et  al. (2008) as an example, this chapter envisions an integrated methodology as an example of multilevel mixing where design choices are guided by an informed understanding of the philosophical and procedural dimensions of each method and thoughtful reflexivity about the

176

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

synergies that can be produced through their pairing. This same logic can be communicated during reporting through a set of interlinked tables and figures (Reeping & Edwards, 2020). The informal group of like-minded colleagues who have been investigating the attributes of MM-GT have each contributed to an advancing understanding of the ways it is being used in practice. Johnson et  al. (2010) launched the conversation by introducing its use in a twophase design, with a qualitative phase devoted to theory development and a quantitative phase prioritizing theory testing. Howell Smith et  al. (2020) illustrated the usefulness of this approach to instrument development. Creamer (2018b) first considered MM-GT as an example of fully integrated mixed method research. Creamer and Schoonenboom (2018) observed that part of the dynamism of mixed method research is the synergistic potential to generate new methodologies that may or may not be unique to the setting where they were used. Guetterman et  al. (2017) and Howell Smith et al. (2020) introduced the first insight that MM-GT is not rare in practice. These two sets of authors have contributed to the literature by beginning to craft and implement a set of guidelines about how to elevate the rigor of studies using MM-GT. Adding to this direction, this commentary has proposed a strategy to systematically embed integration in MM-GT designs.

Implications of a Fully Integrated Approach to MM-GT In the case of MM-GT, the over-arching framework to integration provided in Figure 12.1 and explored further in subsequent tables has implications for the design and execution of this type of research where integration is approached holistically rather than as a set of unrelated procedures. Foremost among these relate to the constant comparative method and the level of interaction between research methods. The comparison of two very different research designs using MM-GT invites further discussion about ways that the constant comparative method can be used with mixed methods by deploying an abductive logic. By detailing an analytic process that begins with induction and moves to abduction, the second, more complex example (i.e., Westhues et  al., 2008) presented in the section about research design does not support Morgan’s view (2020) that abduction replaces induction in a pragmatic framework. This example from Westhues et  al. (2008) adds to Johnson and Walsh’s (2019) argument about how the constant comparative method

can be applied to both qualitative and quantitative data by illustrating how that can be accomplished by merging the data into a single database. Both examples counter the idea proposed by both Johnson and Walsh (2019) and Creamer (2021) that an abductive logic is most often applied during theoretical sampling. The examples demonstrate that the iterative exchange embedded in the constant comparative method can occur after data collection is complete. The contribution of team dynamics to integration in mixed and integrated mixed method designs is a promising area for additional research. It is often noted that one of the challenges faced by investigators using a mixed method approach is that it extends the expectation for methodological expertise to include foundational understanding of both qualitative and quantitative methods. Pairing mixed methods with a qualitative approach, like grounded theory, further intensifies the demand for methodological expertise that is easier to achieve when a team is involved than when undertaken by a single researcher. An informed pairing of mixed methods with grounded theory requires more than a passing acquaintance with the methodological literature about each method. To navigate the publication process and avoid accusations about “method slurring” (Baker et  al., 1992, p. 1355) or what Denzin (2010, p. 420) derides as “methodological poaching,” requires attention to understanding the diversity of perspectives about each research method that moves beyond what can be realistically accomplished in an introductory textbook.

Building an Argument for Methodological Integration Research designs evolve over the life of most research projects. A practical implication of this chapter is to advocate for the benefits of thoughtfully deconstructing the foundational assumptions of methods that are paired in a research project and considering the way they align or misalign before proceeding. While we have proposed this activity as something that might occur as part of the planning process, this is just as likely to be something that proves productive during the reporting phase. An argument for pairing different research methods with mixed methods can provide a solid foundation for a publication that appears in a methodological journal. Articles in a methodological journal are more deeply embedded in the literature about the research method than an empirical article.

BUILDING THE LOGIC FOR AN INTEGRATED METHODOLOGY

Strategies for structuring a rationale for pairing methods that include mixed methods undertaken with a dialectical perspective include: 1 Become familiar with contemporary literature about each methodology and diverse perspectives about it. 2 Organize the manuscript around key dimensions of the argument for the compatibilities between core philosophical assumptions of each method. 3 Acknowledge discontinuities between the core assumptions of the methods and propose ways in which these can be reconciled. 4 Construct a table or figure to summarize similarities and dissonances by the key elements of a methodology. 5 Colour or shading can be used in a figure to communicate similarities and differences between research methods. 6 Be reflexive about how the methodological framework influenced the research process— Table 12.1 is an example of that. 7 Identify elements of the research context, including features of team composition and expertise, that can influence the potential to integrate methodologies in a comprehensive way. 8 If the manuscript is collaboratively produced, demonstrate polyvocality by acknowledging tensions that emerged during the process of interpreting results.

CONCLUSION Fetters and Molina-Azorin (2017) proposed that it is time in the evolution of mixed methods to think more carefully about how we apply the terms “mixing” and “integration”. As compared to mixing or combining, the term “integration” is more appropriate when signaling that integration is approached systematically or with a theoretical perspective that links procedures. This attentiveness to the differences between mixing and integration can extend to the terminology used to explain the pairing of other methods, like grounded theory or participatory action research, with mixed methods. Inter-method mixing is the terminology that communicates a design where methods are paired in a research project but kept relatively distinct. Describing a project as reflecting an integrated methodology, on the other hand, suggests to the reader that considerable effort was invested in the reflexivity required to weigh the

177

compatibilities and tensions between methods and the implications these have to approaching integration in a way that embodies this theoretical perspective.

WHAT TO READ NEXT Creamer, E. G. (2021). Advancing grounded theory with mixed method research. Taylor & Francis/ Routledge.

This book devotes a chapter to exploring steps in the process of using a mixed method approach to core grounded theory procedures. A second chapter explores how tables and figures can be instrumental in theory development. Creamer, E. G., & Schoonenboom, J. L. (2018). Introduction: Inter-method mixing as a gateway to methodological mixing. In E. G. Creamer & J. L. Schoonenboom (Eds.), Methodological innovation in mixed method research. American Behavioral Scientist, 62 (7), 879–886.

This article introduces the idea that new methodologies are constructed when research methods are mixed. Greene, J. C. (2006). Toward a methodology of mixed methods social inquiry. Research in the Schools, 13 (1), 93–98.

In this article, Greene introduces the idea that, as compared to a research method, in social science enquiry, a methodology has four domains (philosophical assumptions and stances, enquiry logic, guidelines for research practice, and sociopolitical commitments).

REFERENCES Apramian, T., Cristancho, S., Watling, C., & Lingard, L. (2017). Re-grounding grounded theory: A close reading of theory in four schools of thought. Qualitative Research, 17(4), 359–376. https://doi. org/10.1177%2F1468794116672914 Baker, C., Wuest, J., & Stern, P. N. (1992). Method slurring: the grounded theory/phenomenology example. Journal of Advanced Nursing, 17, 1355– 1360. https://doi.org/10.1111/j.1365-2648.1992. tb01859.x Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. Sage.

178

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Charmaz, K. (2014). Constructing grounded theory (2nd ed.). Sage. Clarke, A. E. (2007). Grounded theory: Critiques, debates, and situational analysis. In W. Outhwaite & S. Turner (Eds.), The Sage handbook of social science methodology (pp. 423–442). Sage. Creamer, E. G. (2018a). An introduction to fully integrated mixed methods research. Sage. Creamer, E. G. (2018b). Enlarging the conceptualization of mixed method approaches to grounded theory with intervention research. In E. G. Creamer & J. L. Schoonenboom (Eds.), Methodological innovation in mixed method research. American Behavioral Scientist, 62(7), 919–934. Creamer, E. G. (2021). Advancing grounded theory with mixed method research. Taylor & Francis/ Routledge. Creamer, E. G. (2022a). Adding to the understanding of fully integrated mixed methods research. In J. Hitchcock and T. Onwuegbuzie (Eds.), Routledge Handbook for Advancing Integration in Mixed Methods Research (pp. 44–54). Routledge. Creamer, E. G. (2022b). Mixed methods and grounded theory. R. J. Tierney, F. Rizvi, & K. Erickcan (Eds.) International Encyclopedia of Education (4th edition) (pp. 588–566). Science Direct/Elsevier. Creamer, E. G., & Edwards, C. D. (2019). Embedding the dialogic in mixed method approaches to theory development. International Journal of Research and Method in Education, 42(3), 239–251. https://doi. org/10.1080/1743727X.2019.1598357 Creamer, E. G., & Schoonenboom, J. L. (2018). Introduction: Inter-method mixing as a gateway to methodological mixing. In E. G. Creamer & J. L. Schoonenboom (Eds.), Methodological innovation in mixed method research. American Behavioral Scientist, 62(7), 879–886. http://dx.doi.org/ 10.1177/0002764218756917 Denzin, N. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). McGraw Hill. Denzin, N. K. (2010). Moments, mixed methods, and paradigm dialogs. Qualitative Inquiry, 16(6), 419–427. https://doi.org/10.1177%2F1077800410364608 Dey, I. (1999). Grounding grounded theory: Guidelines for qualitative inquiry. Academic Press. Dunne, C. (2011). The place of the literature review in grounded theory research. International Journal of Social Research Methodology, 14(2), 111–124. https://doi.org/10.1080/13645579.2010.494930 Fetters, M. D., & Molina-Azorin, J. (2017). The Journal of Mixed Methods Research starts a new decade: The mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11 (3), 291–307. https://doi.org/ 10.1177%2F1558689817729476 Fetters, M. D., & Molina-Azorin, J. (2019). Rebuttalconceptualizing integration during both data

collection and data interpretation phases: a response to David Morgan. Journal of Mixed Methods Research, 13(1), 12–14. https://doi.org/1 0.1177%2F1558689818780596a Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine. Greene, J. C. (2015). Preserving distinctions within the multimethod and mixed methods research merger. R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 606–625). Oxford University Press. Guetterman, T. C., Babchuck, W. A., Howell Smith, M. C., & Stevens, J. (2017). Contemporary approaches to mixed methods-grounded theory research: A field-based analysis. Journal of Mixed Methods Research, 13(2), 179–195. https://doi. org/10.1177%2F1558689817710877 Guetterman, T. C., Fàbregues, S., & Sakakibera, R. (2021). Visuals in joint displays to represent integration in mixed methods research: A methodological review. Methods in Psychology. https://doi. org/10.1016/j.metip.2021.100080 Hadley, G. (2019). Critical grounded theory. In A. Bryant and K. Charmaz (Eds.), The SAGE handbook of current developments in grounded theory (pp. 564–589). Sage. Hesse-Biber, S. N. (2018). Toward an understanding of a qualitatively driven mixed methods data collection and analysis: Moving toward a theoretically centered mixed methods praxis. In U. Flick (Ed.), The SAGE handbook of qualitative data collection (pp. 545–563). Sage. Howell Smith, M. C., Babchuk, W. A., Stevens, J., Garrett, A., Wang, S. C., & Guetterman, T. G. (2020). Modeling the use of mixed methodsgrounded theory: Developing scales for a new measurement model. Journal of Mixed Methods Research, 14(2), 184–206. https://doi.org/10.1177 %2F1558689819872599 Johnson, R. B., McGowan, M. W., & Turner, L. A. (2010). Grounded theory in practice: Inherently a mixed method? Research in the Schools, 17(2), 65–78. Johnson, R. B., & Onwuegbuzie, A. J. (2007). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26. www.jstor.org/stable/3700093?origin=JSTOR-pdf Johnson, R. B., & Walsh, I. (2019). Mixed grounded theory: Merging grounded theory with mixed methods and multimethod research. In A. Bryant and K. Charmaz (Eds.), The SAGE handbook of current developments in grounded theory (pp. 517–531). Sage. Lillemor R-M., & Hallberg (2006). The “core category” of grounded theory: Making constant comparisons, International Journal of Qualitative Studies on Health and Well-being, 1:3, 141–148. https://doi. org/10.1080/17482620600858399

BUILDING THE LOGIC FOR AN INTEGRATED METHODOLOGY

Kaplan, B., & Duchon, D. (1988, December). Combining qualitative and quantitative methods in information systems research: A case study. Management Information Systems Quarterly, 571–586. https:// doi.org/10.2307/249133 Kushner, K. E., & Morrow, R. (2003). Grounded theory, feminist theory, critical theory: Toward theoretical triangulation. Advances in Nursing Science, 26(1), 30–43. https://doi.org/10.1097/ 00012272-200301000-00006 McCall, C., & Edwards, C. (2021). New perspectives for implementing grounded theory. Studies in Engineering Education, 1(2), 93–107. http://doi. org/10.21061/see.49 McCall, C., McNair, L. D., & Simmons, D. R. (2021). Advancing from outsider to insider: A grounded theory of professional identity negotiation in undergraduate engineering. Journal of Engineering Education, 110(2), 393–413. https://doi.org/ 10.1002/jee.20383 Mertens, D. M., Bledsoe, K. L., Sullivan, M., & Wilson, A. (2010). Utilization of mixed methods for transformative purposes. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social and behavioral sciences (2nd ed., pp. 193–214). Sage. Morgan, D. L. (2020). Pragmatism as the basis for grounded theory. The Qualitative Report, 25(1), 64–73. https://doi.org/10.46743/2160-3715/ 2020.3993 Reeping, D., & Edwards, C. (2020). Advancing 1+1=1 fully integrated designs using five formative figure approach. International Journal of Multiple Research Approaches, 12(3), 1–22. https://doi.org/10.29034/ijmra.v12n3a1

179

Rieger, K. L. (2018). Discriminating among grounded theory approaches. Nursing Inquiry, 26. https:// onlinelibrary.wiley.com/doi/pdf/10.1111/nin.12261 Schwandt, T. A. (2007). Dictionary of qualitative inquiry (3rd ed.). Sage. Sebastian, K. (2019). Distinguishing between the types of grounded theory: Classical, interpretive, and constructivist. Journal for Social Thought, 3(1), 19. Suddaby, R. (2006). From the editors: What grounded theory is not. Academy of Management Journal, 49(4), 633–642. https://doi.org/10.5465/amj. 2006.22083020 Tashakkori, A. & C. Teddlie, C. (Eds.). (2003). SAGE handbook of mixed methods in social and behavioral science research. Sage. Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating qualitative and quantitative approaches in the social and behavioral sciences. Sage. Walsh, I. (2015). Using quantitative data in mixeddesign grounded theory studies: An enhanced path to formal grounded theory in information systems. European Journal of Information Systems, 24, 531–557. https://doi.org/10.1057/ejis. 2014.23 Watson, A. (2019). Methods braiding: A technique for arts-based and mixed methods research. Sociological Research Online, 25(1), 66–83. https://doi. org/10.1177%2F1360780419849437 Westhues, A., Ochocka, J., Jacobson, N., Simich, L., Mailer, S., Janzen, R., & Fleras, A. (2008) Developing theory from complexity: Reflections on a collaborative mixed methods participatory action research study. Qualitative Health Research, 18(5), 701–717. https://doi.org/10.1177/1049732308316531

The Craft of Mixed Methods Research Design: Section 2 Conclusions Judith Schoonenboom and Sophia L. Johnson

In the Introduction to this section, we described mixed methods design as a craft and a product of the interplay between episteme, techne and phronesis. Mixed methods design is not the simple application of general rules, but an iterative and adaptive decision-making process (techne) involving continuous reflection on the ethical implications of those decisions (phronesis). Both these activities are informed by general knowledge (episteme). Individually and collectively, the section chapters have demonstrated the importance of a craft approach to mixed methods research design. In Chapter 12, Creamer, McCall and Edwards warn against the unreflected adoption of grounded theory coding without considering its embeddedness in a wider logic of inductive theory building. Suboptimal designs can also result from unreflectively sticking to previous decisions. Chapter 7, De Allegri and Lohmann demonstrate that it makes sense to reconsider decisions when societal and policy concerns change, additional data sources become available, or unexpected findings emerge. Collectively, the section chapters provide much of the knowledge and skills needed for making informed decisions in mixed methods research. They cover a substantial part of the mixed methods research process: emergence (Chapter 7, De Allegri & Lohmann), visualization (Chapter 8,

Schoonenboom), combination with other methodologies (Chapter 12, Creamer, Edwards & McCall), sampling (Chapter 9, Corrigan & Onwuegbuzie), integrated data analysis (Chapter 10, Vogl) and ethics (Chapter 11, Cameron & Herrmann). The chapters contain four important take-away messages, which will be discussed in more detail below: • Design decisions are not simple and global but complex, multi-step, detailed and iterative. • The mixed methods design process is permeated by ethical considerations. • The mixed methods design process involves integration at many levels. • Several heuristics and tools can help mixed methods researchers in navigating design decisions.

DESIGN DECISIONS ARE NOT SIMPLE AND GLOBAL BUT COMPLEX, MULTISTEP, DETAILED AND ITERATIVE In mixed methods research, design decisions are subject to reconsideration based on changes in the course of the project—in other words, mixed methods design is emergent. In Chapter 7, De Allegri

SECTION 2 CONCLUSIONS

and Lohmann show that design decisions are iterative and are reconsidered throughout the study, while designing, during sampling and data collection, and during data analysis and interpretation. In Chapter 10, Vogl provides an example of iteration in data analysis. Design decisions in mixed methods research are complex. Sometimes, the needed episteme (general knowledge) is complex. In Chapter 11, Cameron and Herrmann show that mixed methods research using big data requires knowledge from various related fields. They warn against the unreflective and uninformed use of tools that have been developed to make big data analysis easier. In Chapter 12, Creamer, Edwards and McCall demonstrate the complexity of research decisions when combining mixed methods research with other methodologies. Successful combination requires scrutiny of compatibility and potential synergies at various levels of mixing. Design decisions are made both at a global and a detailed level. In Chapter 8, Schoonenboom shows how unraveling research strands at a detailed level enables researchers to reflect the fit of design elements in more detail, reveals hidden design decisions and makes visible the many different types of integration among research strands. Lastly, mixed methods design decisions involve multiple steps. In Chapter 9, Corrigan and Onwuegbuzie show that sampling does not “simply” mean selecting participants either at random or purposefully; it involves reflecting on paradigmatic assumptions, study purpose, populations, approaches, design types and relationships among research strands.

THE MIXED METHODS DESIGN PROCESS IS PERMEATED BY ETHICAL CONSIDERATIONS According to Cameron and Herrmann, ethical issues must be considered during the entire mixed methods research process. Ethics is especially important in a craft approach to mixed methods research because its multiple and iterative design decisions each have an ethical component, as each will affect practice. De Allegri and Lohmann point out that emergent designs pose a challenge to ethics committees. Committee members cannot be expected to approve beforehand possible changes whose ethical implications they cannot judge. In sampling, ethics is about whom/what we include and whom/what we exclude. Corrigan and Onwuegbuzie

181

distinguish between virtue ethics (referring to the character of the mixed methods researcher as providing the impetus for ethical sampling behavior) and pragmatic ethics (using the ethical standards established by communities). Cameron and Herrmann discuss five ethical criteria for mixed methods research with big data: beneficence (e.g., privacy), non-malevolence (e.g., cybercrime), justice (e.g., rectifying bias and inequality), explicability (research that is understandable in human language) and autonomy (of humans over artificial intelligence).

THE MIXED METHODS DESIGN PROCESS INVOLVES INTEGRATION AT MANY LEVELS The chapters in this section have highlighted the importance of integration. Creamer, Edwards and McCall have shown how two or more research approaches with complementary assumptions can be meaningfully integrated. This integration occurs at various levels, from the methodologies’ underlying paradigms to how results are reported. Corrigan and Onwuegbuzie relate the complexity of sampling to the level of integration of qualitative and quantitative components in a study: Fuller integration of qualitative and quantitative components requires integrated sampling designs that are potentially more complex. De Allegri and Lohmann describe how embracing emergence enhances integration across quantitative and qualitative findings, leading to integration beyond and different from the integration points formulated beforehand. The detailed methods-inference map in Schoonenboom reveals many integration points that would have remained hidden in a less detailed approach. It also displays the integration between methods and inferences, and shows how the meta-inference is an emergent result of successive integrations. This section has elucidated three aspects of data integration: using data for various purposes, data conversion and returning to data already analyzed (see also Creswell & Plano Clark, 2018; Schoonenboom, 2023). Vogl emphasizes the advantages of collecting qualitative and quantitative data from the same participants. After data conversion (quantitizing the qualitative data), each participant’s quantitative and quantitized qualitative data can be combined into a new data set. Thus, data can be linked directly, something which is impossible when data stem from independent samples or different aggregate levels.

182

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

De Allegri and Lohmann emphasize the importance of returning to the data for further analysis during a study. This reuse of data is made visible in Schoonenboom’s methods-inference map.

SEVERAL HEURISTICS AND TOOLS CAN SUPPORT MIXED METHODS RESEARCHERS IN NAVIGATING DESIGN DECISIONS This section has demonstrated that design decisions are much more numerous and complex than may appear at first sight. To navigate these many complex design decisions, the authors of this section have shown how existing mixed methods design concepts can be applied in an innovative way or a new context. Cameron and Herrmann have shown how five ethical criteria apply to mixed methods research that uses big data. Vogl has demonstrated how the challenge of data integration can be tackled by combining three data integration types—linkage, transformation and consolidation in an innovative way. Several chapters present models that support the design process. De Allegri and Lohmann’s table (7.1) supports mixed methods researchers in thinking about how they can embrace emergence throughout the mixed methods research process. Creamer, Edwards and McCall’s multilevel framework supports mixed methods researchers in meaningfully combining methodologies by identifying their potential synergies at various levels. Schoonenboom’s methods-inference map is a visual model that supports researchers in reflecting on their research components and their interactions at both a global and a detailed level, showing how methods and inferences interact throughout a study as the meta-inference develops. Visually most outstanding is Corrigan and Onwuegbuzie’s tree sampling meta-framework, with its roots, trunk, limbs, branches, twigs and leaves, which is further elaborated in their flowchart. The take-away messages of this chapter can be summarized in five slogans. “Be flexible” applies to data integration (Chapter 10, Vogl); emergence (Chapter 7, De Allegri & Lohmann) and choosing findings for further exploration (Chapter 8, Schoonenboom). Flexibility is also visible in De Allegri and Lohmann’s slogan, “Follow the thread,” which should be accompanied by “Return to your data” (Chapter 7, De Allegri & Lohmann; Chapter 10, Vogl). Flexibility, though, should be kept in check. “Check the fit” has been the main principle of Maxwell’s (2013) interactive design

model, discussed by De Allegri and Lohmann and by Schoonenboom. Checking the fit also applies to fundamental assumptions when combining mixed methods with another methodology (Chapter 12, Creamer, Edwards & McCall). We can only check the fit between elements that we have identified. Schoonenboom has demonstrated that it makes sense to identify elements at a detailed level, leading to the last slogan: “One research strand for each research question and each population.”

IMPLICATIONS What are the implications of viewing mixed methods design as a craft and a product of the interplay between episteme, techne and phronesis? Knowledge-in-context (techne), or the ability to make adaptive and informed decisions, can only be developed through practice and exposure to various contexts (Sanscartier, 2020). The chapters in this section support mixed methods researchers in developing this knowledge-in-context by providing heuristics, tools and principles that guide decisionsin-context, including principles for making ethical decisions (Sanscartier’s phronesis). What episteme—general knowledge—should mixed methods researchers have? The authors of this section have pointed out that the multi-step and embedded character of design decisions requires different types of general knowledge. Thus, mixed methods researchers need knowledge about foundational paradigms and world views (Chapter 9, Corrigan & Onwuegbuzie; Chapter 12, Creamer, Edwards & McCall), various types of research questions and forms of data analysis (Chapter 10, Vogl), sampling (chapter 9, Corrigan & Onwuegbuzie), and validity criteria (Chapter 11, Cameron & Herrmann; Chapter 7, De Allegri & Lohmann). Further, researchers need general knowledge of the methodologies or research designs included in their mixed methods study, including their assumptions. Research designs, such as the experiment, the case study, ethnography or grounded theory, often have a tradition of use of decades and they provide guidance in many design decisions (Chapter 9, Corrigan & Onwuegbuzie; Chapter 12, Creamer, Edwards & McCall). Sanscartier’s (2020) episteme does not refer to general knowledge of existing research designs or methodologies, but to general knowledge in the form of mixed methods typologies: In the specific field of mixed methods research, episteme most closely corresponds to typologies

SECTION 2 CONCLUSIONS

routinely applied to various research contexts . . . They … constitute a family of context-independent strategies to carry out mixed methods designs, potentially guiding the timing and purpose of data integration (e.g., sequential vs. concurrent designs). (p. 51)

Among the authors of this section, opinions differ as to the role of mixed methods design typologies. The choice between a sequential or concurrent design, mentioned by Sanscartier (2020), is one of the choices in Corrigan and Onwuegbuzie’s tree sampling meta-framework. More importantly, Corrigan and Onwuegbuzie recommend using the Leech and Onwuegbuzie (2009) threedimensional typology for research designs with the dimensions level of mixing, time orientation and emphasis of approaches (e.g., a fully mixed sequential equal status design). In the other chapters, design typologies are largely absent. Vogl notes that the possibilities for data integration are not affected by whether the design is concurrent or sequential, or whether data are collected once or multiple times. Instead, a crucial question for data integration is whether the data have been obtained from the same participants and can therefore be linked directly, or whether they come from independent samples or different aggregate levels and can thus only be linked at a conceptual level.

183

(see Hesse-Biber & Johnson, 2015). Cameron and Herrmann point out that big data and its societal impact will become data sources for future mixed methods research. As analysis will change from batch to real-time analysis, mixed methods researchers will have to acquire the needed ICT skills. Creamer, Edwards and McCall’s model based on grounded theory could in the future be extended to combinations of other methodologies with mixed methods (see also Creamer & Schoonenboom, 2018). Lastly, Schoonenboom’s methods-inference map reveals the importance of subgroup analysis in mixed methods research, raising the question of the circumstances under which subgroup analyses could (or perhaps even should) be performed (see also Schoonenboom, 2019). The methods-inference map also shows the stepwise development of the meta-inference (Schoonenboom, 2022), raising the question of how (i.e., using what development steps in what order) a meta-inference can best be developed. Tomorrow cannot be predicted, but one thing is clear: the future is for mixed methods as a craft, an iterative and adaptive decision-making process, and a continuous practice of reflection on the ethical implications of research choices—informed, but not dictated, by general knowledge.

REFERENCES IMAGINING THE FUTURE OF MIXED METHODS RESEARCH DESIGN When mixed methods researchers start to apply this section’s mixed methods heuristics and tools, they will have a broad impact on their research practice. According to Corrigan and Onwuegbuzie, applying a transparent, rigorous, equitable and ethical matching process will contribute to procedural justice by creating space for (and appropriate representation of) the voices of underrepresented and marginalized groups. De Allegri and Lohmann’s hope for the future is a more open attitude of institutions and funding agencies toward emergent designs. Heuristics and tools are never complete, which raises the question of their future development. Vogl recommends broadening the integration debate to the integration of any type of data or perspective. This recommendation aligns with the current trend toward the inclusion of multimethod research (involving a combination of several qualitative or several quantitative approaches) as a topic of interest to the mixed methods community

Creamer, E. G., & Schoonenboom, J. (2018). Intermethod mixing as a gateway to methodological innovation. American Behavioral Scientist, 62(7), 879– 886. https://doi.org/10.1177/0002764218756917 Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Hesse-Biber, S., & Johnson, R. B. (Eds.). (2015). The Oxford handbook of multimethod and mixed methods research inquiry. Oxford University Press. Leech, N. L., & Onwuegbuzie, A. J. (2009). A typology of mixed methods research designs. Quality and Quantity, 43(2), 265–275. https://doi.org/ 10.1007/s11135-007-9105-3 Maxwell, J. A. (2013). Qualitative research design: An interactive approach (3rd ed.). Sage. Sanscartier, M. D. (2020). The craft attitude: Navigating mess in mixed methods research. Journal of Mixed Methods Research, 14(1), 47–62. https:// doi.org/10.1177/1558689818816248 Schoonenboom, J. (2019). Develop your case! How controversial cases, subcases, and moderated cases can guide you through mixed methods data analysis. Frontiers in Psychology, 10(1369). https:// doi.org/10.3389/fpsyg.2019.01369

184

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Schoonenboom, J. (2022). Developing the metainference in mixed methods research through successive integration of claims. In J. H. Hitchcock & A. J. Onwuegbuzie (Eds.), The Routledge handbook for advancing integration in mixed methods research (pp. 55–70). Routledge. https://doi. org/10.4324/9780429432828-6

Schoonenboom, J. (2023). Ten mixed methods integration strategies for obtaining a detailed understanding. In R. Tierney, F. Rizvi, K. Ercikan, & G. Smith (Eds.), International encyclopedia of education (4th ed.). Elsevier. https://doi.org/10.1016/ B978-0-12-818630-5.11045-0

SECTION 3

Expanding Mixed Methods Design Approaches

This page intentionally left blank

Expanding Beyond Typology-Based Mixed Methods Designs: Section 3 Introduction P e g g y S h a n n o n - B a k e r a n d J e s s i c a T. D e C u i r - G u n b y

As Section 1 of the Handbook explores, typologies for mixed methods research designs have developed over time (cf. Creswell & Plano Clark, 2017). Typologies initially helped to legitimize mixed methods research as a methodology because they provided an organizing framework for how to describe, design, and do mixed methods research (Tashakkori et  al., 2021). Using a typology to teach mixed methods research to emerging researchers can help them envision how to apply the methodology (Ivankova & Plano Clark, 2018; Onwuegbuzie et al., 2011; Poth, 2014; Tashakkori et  al., 2021; see also Chapter 32, this volume). However, typologies can be limiting. These frameworks can imply a sense of linearity in the design and implementation process whereas mixed methods research is often emergent and messy (Bazeley, 2003; Brevik, 2022; Poth, 2020; Sakata, 2022; Walton et al., 2020). The purpose of Section 3—“Expanding Mixed Methods Design Approaches”—is to expand beyond typology-based mixed methods designs. We solicited chapters within this section to expand typologies of mixed methods research based on two key influences: cultural contexts and the use of qualitative designs (e.g., grounded theory) or quantitative designs (e.g., randomized control trials). Our interest in working together on this section came from the potential we see in using

the power of culture and design combinations to explore innovative applications of mixed methods research. Our aim in this section is not to present a new typology of mixed methods designs, but rather to showcase a sample of how mixed methods researchers meaningfully respond to cultural influences and other design types in their work. For further discussions of cultural influences in mixed methods design, see also Section 5, Introduction (this volume).

ORGANIZATION OF THE SECTION We organized the nine chapters in this section within two themes: cultural considerations in mixed methods designs and mixed methods designs that are combined with design types from other methodologies. The authors provide foundational information for those contexts and design features, one or more illustrative example studies that demonstrate their design in practice, and a discussion of the implications for their unique design for mixed methods research. The authors in this section discuss their work in the Caribbean, Malaysia, New Zealand, Samoa, South Africa, the UK, and the USA. Their disciplinary backgrounds

188

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table S3.1.  Summary of Section 3 chapters: Expanding Mixed Methods Design Approaches Chapter authors (country affiliation)

Chapter title

Jenny Douglas (UK)

Exploring Interlocking Relationships of Race, Gender, and Class with an Intersectionality-Informed Mixed Methods Research Design Framework Indigenous Cultural Values Instrument Development: Using Mixed Methods Research What Can Mixed Methods Partnerships Learn from Kaupapa M¯aori Research Principles? Prioritizing Cultural Responsiveness in Mixed Methods Research and Team Science with Underrepresented Communities

Mehdi Taghipoorreyneh (Iran) Peter Rawlins, Philippa Butler, Spencer Lilley, and Maggie Hartnett (New Zealand/Aotearoa) Tera R. Jordan and Maya Bartel (USA) Jenevieve Mannell and Audrey Prost (UK) Joanne Mayoh (UK), Talia Thompson (USA), and Shanlee Davis (USA) Loraine D. Cook (Jamaica) and Vimala Judy Kamalodeen (Trinidad & Tobago) Vanessa Scherman and Lisa Zimmerman (South Africa) Michelle C. Howell, Wayne A. Babchuk, and Timothy C. Guetterman (USA)

Using Participatory Methods in Randomised Controlled Trials of Complex Interventions Illustrating the Mixed Methods Phenomenological Approach (MMPR) Intersection of Mixed Methods and Case Study Research (MM+CSR): Two Design Options in Educational Research Harnessing Mixed Methods for Research Instrument Development and Legitimation Mixed Methods-Grounded Theory: Best Practices for Design and Implementation

are similarly wide ranging, including education, health promotion, business and management, and health sciences. Table S3.1 lists the chapter authors and their titles for Section 3.

Cultural Considerations in Mixed Methods Designs The first four chapters in this section center the importance and influence of cultural contexts on the mixed methods process. The chapters in this theme largely describe how cultural considerations influence broader elements of the mixed methods process, including: collaborating with researchers, participants, and community partners (Chapter 14, Taghipoorreyneh; Chapter 15, Rawlins et  al.; Chapter 16, Jordan & Bartel); using building integration procedures for ongoing feedback (Chapter 14, Taghipoorreyneh); and aligning elements of the mixed methods process with relevant philosophical and theoretical frameworks (Chapter 13, Douglas; Chapter 15, Rawlins et  al.). In Chapter 13, Douglas explores what mixed methods research can look like when its design and application is informed by intersectionality, which is a theory that explores how policies, systems, and structures further discriminate against people who are multiply marginalized (Crenshaw, 1991). When applied to research practices, intersectionality can be used to inform

research questions, how researchers analyze data, and how they interpret data (Bowleg, 2008). Douglas’s chapter draws from an example study that explored the intersections of race, gender, and socioeconomic class in a study about cigarette smoking among African-Caribbean teen girls. Whereas many cultural values instruments tend to be based on Western values, Taghipoorreyneh’s chapter (Chapter 14) details his application of a three-round Delphi study in creating an Indigenous cultural values instrument based on the largest cultural group in Malaysia: Malay. Originally developed to make future projections, the Delphi method is also used in research to test questionnaire items through multiple rounds of feedback with key stakeholders and experts. Taghipoorreyneh used this method to get feedback on the cultural appropriateness of the items in his instrument and, finally, he used correspondence analysis of scale to ensure the items represented this cultural group in Malaysia. In Chapter 15, Rawlins and colleagues similarly explore the influence of cultural values on the mixed methods research process, focusing on the principles of Kaupapa M¯aori research. Kaupapa M¯aori research centers M¯aori people, culture, knowledge, and language: “It is undertaken by M¯aori. It is for M¯aori and it is with M¯aori” (Smith, 2015, p. 47). Contextualized in a broad history of New Zealand, their chapter outlines three research frameworks with increasing focus and centering of M¯aori people and values: research involving

SECTION 3 INTRODUCTION

M¯aori, M¯aori-centered research, and Kaupapa M¯aori research. They then discuss several example mixed method studies that showcase these research frameworks to varying degrees. In Chapter 15, Rawlins and colleagues describe the importance of researcher positionality within M¯aori communities or identification as M¯aori. In Chapter 16, Jordan and Bartel then employ similar reflexive positioning to a broader level, looking at the varying nature of interdisciplinary, multidisciplinary, and transdisciplinary mixed methods teams that work in or with marginalized communities. They briefly summarize many example studies to discuss how each team navigated challenges in the team dynamics, research process, and sharing research findings.

Combination Mixed Methods Designs Several other chapters in Section 3 also describe the influences of cultural contexts in their example studies. These chapters, however, emphasize the influence of combining mixed methods with a qualitative or quantitative design. For example, in Chapter 17, Mannell and Prost describe the importance of using participatory methods in a randomized controlled trial (RCT) to evaluate a program designed to prevent violence against women in Samoa. Participatory RCT allowed them to design and use culturally appropriate methods and engage stakeholders in interpreting the study results. In Chapter 18, Mayoh and colleagues demonstrate how phenomenologically driven mixed methods can be used to bring in underrepresented voices in the research process. They draw from their own research with young women with Turner Syndrome, where females are missing part or all of the second sex chromosome, through clinics and support groups in Colorado, USA. Additionally, Cook and Kamalodeen’s chapter (Chapter 19) outlines the difference between case study mixed methods and mixed methods case study. This design-based combination allowed them to explore Caribbean teacher educators’ experiences moving online due to COVID-19. While written together, this chapter was published posthumously after Vimala Kamalodeen’s sudden passing in 2022. Whereas Taghipoorreyneh’s chapter emphasizes the importance of allowing for cultural nuances to be represented in the instrument development process, Scherman and Zimmerman (Chapter 20) take a broader perspective on instrument development in mixed methods research. They outline how instruments for data collection are developed in quantitative and qualitative

189

research. Then they describe example studies that use integration procedures to create instruments based on data synthesis, combination, or transformation. Finally, in Chapter 21, Howell and colleagues describe several key features of a mixed methods grounded theory (MM-GT) design based on a systematic review: writing MM-GT oriented research questions, using MM-GT related references, and substantiating the rationale, design elements and rigor of the study.

CROSS-CUTTING DISCUSSIONS AMONG THE SECTION 3 CHAPTERS Collectively, the chapters in Section 3 address scale development, the importance of reflexive practices, and the use of more qualitatively driven approaches in mixed methods research. The chapters from Taghipoorreyneh, and Scherman and Zimmerman showcase how scale development is a mixed methods process. These two chapters present different perspectives to consider when using mixed methods for instrument development. Where Jordan and Bartel argue for the importance of research teams engaging in reflexive practices when conducting mixed methods research, other chapters in this section also illustrate this importance, such as the extent to which non-M¯aori researchers can engage in various forms of M¯aoriinformed research practices (Chapter 15, Rawlins et al.), the use of postpositivist methods of reflection such as bracketing (Chapter 18, Mayoh et  al.), and using researchers’ own reflections as another source of data (Chapter 19, Cook & Kamalodeen). Overall, the use of these combined designs and participatory practices highlights the increasing interest in qualitatively oriented mixed methods research (cf. Poth & Shannon-Baker, 2022; Toledo & Shannon-Baker, in press). Three chapters in our theme about combining mixed methods with other research designs explored this combination with traditionally qualitative designs: grounded theory (Chapter 21, Howell et  al.), case study (Chapter 19, Cook & Kamalodeen) and phenomenology (Chapter 18, Mayoh et  al.). Several chapters throughout the section also demonstrate the value of engaging in participatory methods with research participants, team members, and stakeholders during the mixed methods process (Chapter 16, Jordan & Bartel; Chapter 17, Mannell and Prost; Chapter 18, Mayoh et  al.; Chapter 14, Taghipoorreyneh; Chapter 15, Rawlins et al.). While reading the chapters in this section, readers will gain a foundation in the various design

190

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

approaches and cultural contexts discussed, an understanding of challenges and benefits to these approaches, and ideas to further innovate in their own uses of mixed methods designs.

REFERENCES Bazeley, P. (2003). Teaching mixed methods. Qualitative Research Journal, 3, 117–126. Bowleg, L. (2008). When Black + lesbian + woman ≠ Black lesbian woman: The methodological challenges of qualitative and quantitative intersectionality research. Sex Roles, 59(5–6), 312–325. https:// doi.org/10.1007/s11199-008-9400-z Brevik, L. M. (2022, April). Planned and emergent designs: The value of planning for the unexpected in mixed-methods research. American Educational Research Association Annual Meeting, San Diego, CA. Crenshaw, K. (1991). Mapping the margins: Intersectionality, identity politics, and violence against women of color. Stanford Law Review, 43(6), 1241–1299. https://doi.org/10.2307/1229039 Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). Sage. Ivankova, N. V., & Plano Clark, V. L. (2018). Teaching mixed methods research: Using a socio-ecological framework as a pedagogical approach for addressing the complexity of the field. International ­Journal of Social Research Methodology, 21(4), 409–424. https://doi.org/10.1080/13645579.201 8.1427604 Onwuegbuzie, A. J., Frels, R. K., Leech, N. L., & Collins, K. M. (2011). A mixed research study of pedagogical approaches and student learning in doctorallevel mixed research courses. International Journal of Multiple Research Approaches, 5(2), 169–199. https://doi.org/10.5172/mra.2011.5.2.169

Poth, C. (2014). What constitutes effective learning experiences in a mixed methods research course? An examination from the student perspective. International Journal of Multiple Research App­ roaches, 8(1), 74–86. https://doi.org/10.5172/mra. 2014.8.1.74 Poth, C. N. (2020). Confronting complex problems with adaptive mixed methods research practices. Caribbean Journal of Mixed Methods Research, 1(1), 29–46. www.mmiracc.com/_files/ugd/01fe3f_10c691 8eb80c49e7976d80536c6bdd50.pdf#page=49 Poth, C. N., & Shannon-Baker, P. (2022). State of the methods: Leveraging design possibilities of qualitatively oriented mixed methods research. International Journal of Qualitative Methods, 21, 1–11. https://doi.org/10.1177/16094069221115302 Sakata, N. (2022). Embracing the messiness in mixed methods research: The craft attitude. Journal of Mixed Methods Research, Online First. https://doi. org/10.1177/15586898221108545 Smith, L. T. (2015). Kaupapa M¯aori research – Some Kaupapa M¯aori principles. In L. Pihama & K. South (Eds.), Kaupapa Rangahau a reader: A collection of readings from the Kaupapa M¯aori Research Workshop Series Led (pp. 46–52). Te Kotahi Research Institute. https://researchcommons.­ waikato.ac.nz/handle/10289/12026 Tashakkori, A., Johnson, R. B., & Teddlie, C. (2021). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences (2nd ed.). Sage. Toledo, C. & Shannon-Baker, P. (in press). Choosing a qualitatively oriented mixed methods research approach: Recommendations for researchers. In R. Cameron & X. Golenko (Eds.), Handbook of mixed methods in business and management. Edward Elgar. Walton, J. B., Plano Clark, V. L., Foote, L. A., & Johnson, C. C. (2020). Navigating intersecting roads in a mixed methods case study: A dissertation journey. Journal of Mixed Methods Research, 14(4), 436–455. https://doi.org/10.1177/1558 689819872422

13 Exploring Interlocking Relationships of Race, Gender, and Class with an IntersectionalityInformed Mixed Methods Research Design Framework Jenny Douglas INTRODUCTION The aim of this chapter is to explore how intersectionality-informed research applies to mixed methods research. This chapter draws from a study about African-Caribbean young women and cigarette smoking in the UK. There is a need for novel understandings of the interlocking relationships of race, gender and class to inform policy (Hankivsky & Jordan-Zachary, 2019). There is limited research on the health behaviours and experiences of Black and minority ethnic communities, as existing research focuses on White communities in the UK (Douglas, 2014). It is important to understand how the categories of race, gender and class intersect, shape and influence each other and influence the health of Black and minority ethnic people (Bowleg, 2012). My particular interest is research on gender, ethnicity and health. This is an area that often fails to reflect the complexity of the lives of Black and minority ethnic groups, and to contextualize them within historic, social and socioeconomic circumstances, or power and sociostructural contexts. An intersectionality-informed approach to research allows us to examine the dynamic processes influencing an individual’s experiences and enables a more nuanced examination of their impact on individuals and communities. Mixed

methods approaches can integrate structural-level and multilevel quantitative approaches with qualitative approaches that can explore complex meanings. Therefore, my development and use of an intersectionality-informed mixed methods design allowed me to generate novel insights that were important for my area of study. In this chapter, I advance a framework for guiding researchers’ development of intersectionalityinformed mixed methods research interweaving discussions of an illustrative study of the social ­phenomenon of cigarette smoking. The significance of an intersectionality-informed mixed methods research design is its ability to capture the complex interactions between structures of power and oppression, and interconnected aspects of individual and group identity and social location (Grace, 2014, p. 1). While the quantitative data represents health behaviours, the integration of qualitative data captures experiences and perspectives, and this produces new insights previously inaccessible by either data type alone. I begin with situating the potential for an intersectionality-informed mixed methods research design for generating novel understandings of the interlocking gender and race influences on health behaviours, and make the case for a guiding MMR design framework with distinctive features. Then

192

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

I introduce the background to and design of my illustrative study using a convergent mixed methods research design and interweave discussions of this completed study into a description of a framework for guiding the development. Finally, I offer concluding thoughts about how the intersectionality-informed framework helps mixed methods researchers navigate the challenges experienced in intersectional research and identify future directions for researchers to pursue.

SITUATING THE POTENTIAL OF INTERSECTIONALITY-INFORMED MIXED METHODS RESEARCH An Introduction to Intersectionality Research Kimberlé Crenshaw coined the term “intersectionality” more than thirty years ago (Crenshaw, 1989). She offered intersectionality theory as a mechanism for examining the complex intersection between race and gender and other determinants that marginalized Black women’s experiences and treated “race and gender as mutually exclusive”. Crenshaw (1991) argued that women of colour face heightened discrimination because of their social location at the intersections of marginality based on race and gender, as well as other locations such as class, language, and nationality. African American social scientists had critiqued second-wave feminism which privileged gender, but did not give any significance to race or ethnicity. Intersectionality was originally developed as a legal theory (Crenshaw, 1989). However, in many different disciplines and fields in social sciences and humanities, it is now used as an organizing category for feminist enquiry (Lewis, 2013). Lewis (2013) argues that intersectionality can be conceived as a theory, concept, methodology, heuristic—or, in fact, all four. I use it as all four. Intersectionality has been applied as a theory to explore inequalities and inequity in health (Bowleg, 2012; Hankivsky, 2012). McCall (2005) argued that although intersectionality has emerged as a major paradigm in research, there had been limited discussion of how an intersectional methodology might work. Intersectionality is a complex phenomenon, and it is very difficult to capture complexity in social research. It can be argued that intersectionality makes claims about interrelationships between different categories (i.e., anti-, intra- and inter-complexity), but intersectional researchers rarely show how these categories are

interrelated. McCall (2005) puts forward three approaches for the study of multiple, complex social relations: anti-categorical complexity, intracategorical complexity, and inter-categorical complexity. Anti-categorical complexity deconstructs analytical categories on the basis that social life is too complex to reduce to fixed categories. In terms of intra-categorical complexity, researchers tend to focus on particular social groups at neglected points of intersection, while regarding inter-categorical complexity, scholars adopt existing analytical categories to explore inequality. My illustrative study uses an intra-categorical approach to intersectionality. It seeks to highlight diversity within groups (i.e., young Black women) by taking marginalized intersectional identities as an analytic starting point in order to reveal the complexity of lived experiences within such groups (McCall, 2005, p. 1774). One of the challenges of my illustrative study is to explore the identities of African-Caribbean young women and consider how this is related to cigarette smoking, as this was previously unexplored. This research may require the development of a theoretical conceptualization of identity which draws on both modernist and postmodern perspectives—that is, one that recognizes that identity is influenced by race, ethnicity, gender, social class, sexual orientation and the oppression that is associated with these factors through power relationships in society, while at the same time acknowledging that identity is not fixed but changing and fluid. Hence, rather than regarding agency and structure as polar binaries, a framework is needed that incorporates agency and structure in a complementary rather than oppositional way. An intersectional approach can do this (Collins, 1990; Crenshaw, 1989, 1991) by recognizing the simultaneous influence of race, gender and ethnicity and the impact of these on social structures and axes of oppression. Mirza talks of the notion of “embodied intersectionality”, which refers to the lived experiences of Black women (Mirza, 2009).

Existing Approaches and Remaining Gaps/Influences? The aim of my illustrative study was to examine the cultural, historical and social contexts of cigarette smoking. In order to understand cigarette smoking, or any other health behaviour, it is important to understand the cultural context. When working with an anti-racist perspective, it is necessary that Black communities are involved throughout the research process from the development of the research proposal, through to the execution of

EXPLORING INTERLOCKING RELATIONSHIPS OF RACE, GENDER AND CLASS

the research, the research analysis and the dissemination of research. While I have sustained this particular lens in relation to research methodologies and have developed an intersectional approach, which incorporates anti-racism, it was not until recently that anti-racist research practice was accorded a high priority by members of the academic research community. However, the death of George Floyd in May 2020 served to bring about a seismic shift in terms of the centring of anti-racist research practice in a range of disciplines and methodologies. Variables such as race and ethnicity, class, disability, sexual orientation and gender (among others), are often viewed as independent, and social research methodologies do not examine their interactions. “Intersectionality” is an approach that recognizes that these dimensions are not isolated, independent variables that are simply “additive”, but rather that they are interlocking and interactive (Douglas, 2020). An intersectionality-informed approach to research allows us to examine the dynamic processes influencing individuals’ experiences, and enables a more nuanced examination of their impact on individuals and communities. When researching the health behaviours of Black and minority ethnic communities, it is important to reflect the complexity of their lives, and to contextualize them within historical, social and socioeconomic circumstances, of power and sociostructural contexts. It is important to consider power and its role in structuring and reinforcing social categories in order to understand its impact on health and health behaviour such as cigarette smoking. These processes and issues are dynamic and changing so that, as health experiences of Black women change, so do the impact of inequities on their health (Douglas, 2018; Douglas & Watson, 2013). There are continuing debates about the ontological and epistemological differences between qualitative and quantitative paradigms in discussions of health research and beyond. This has implications for how data can be combined in mixed methods to generate novel understandings of complex research problems. Hughes and Cohen argue that methods should be chosen that are most appropriate to addressing the research question and therefore “challenge the simplistic, and consequential, presumption that to do feminist empirical research one has to use qualitative methods” (2010, p. 190). The discussions about quantitative and qualitative methods were in turn critiqued by Black feminist researchers for ignoring the interaction between race class and gender (Collins, 1991; Crenshaw, 1989; Davis, 1981; Mullings, 2000). Developed by Black feminists, intersectionality theory tries to address the complexity of social life

193

by recognizing that individuals simultaneously occupy multiple social locations. However, trying to develop an intersectional approach for social research presents a challenge. Denis (2008) comments that the practice of developing appropriate intersectional methodologies has not caught up with the theory.

Distinguishing Features of Intersectionality-Informed Mixed Methods Research Intersectionality-informed mixed methods research enables researchers to develop novel understandings of the interlocking relationships of race, gender and class. Hankivsky and Grace (2015) argue that the relationship between intersectionality and mixed methods is in nascent stages of development. Despite this, they suggest that, while intersectionality does not prescribe any particular research method or design, an intersectionality-informed stance can benefit any research strategy by incorporating and integrating the principles of intersectionality. Furthermore, intersectionality can encourage researchers to pay attention to the needs of understudied populations. Bowleg (2008) notes that whether researchers are using qualitative or quantitative research methods, they bear “the responsibility for interpreting their data within the context of sociohistorical and structural inequality” (p. 321). Hankivsky and Grace (2015) also caution researchers about undertaking intersectionalityinformed research without the necessary knowledge and understanding. Such research offers unique and transformative opportunities to understand social difference, inequity and power. They argue that as well as having in-depth knowledge of the intersectionality research literature, researchers also require a detailed understanding of mixed methods research. Although this field is in the early stages of development, there are growing and developing resources available (Hankivsky & Grace, 2015). Furthermore, Cole (2009) purports that an intersectionality framework asks researchers to examine categories of identity, difference and disadvantage with a new lens (p. 170), and she outlines a series of three questions that researchers can ask to conceptualize the influences of multiple social categories. Cole (2009) proposes that researchers adopting an intersectional approach should ask the following three questions: Who is included within this category? What role does inequality play? Where are there similarities?’ (p. 172). Using these three questions, I analyze

194

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

my findings and the role that disadvantage plays in the lives of African, African-Caribbean and White young women in my study, paying particular attention to the differences and similarities. In Table 13.1, I adapt Cole’s approach to intersectionality-informed Mixed Methods Research and suggest how researchers can ensure how matters of social justice can meaningfully be addressed so that research does not reproduce existing inequalities. This table examines how individual researchers can apply Cole’s three questions to their own research. To relate to existing mixed methods research literature, Shannon-Baker (2021) undertook a systematic review of mixed and multi-method studies in implicit bias and race, and Onwuegbuzie et al. (in press) have developed IMAGINE–Integrative,

Mixed methods, Anti-racist Groundwork for Investigating and Nurturing Equity. These examples of using intersectionality with mixed methods research demonstrate how issues of social difference, inequity and power can be reimagined.

ORIENTATION TO THE ILLUSTRATIVE STUDY OF CIGARETTE SMOKING Although African-Caribbean young people form a relatively high percentage of young people in many British cities, there is limited research on their health and on cigarette smoking. African and African-Caribbean young women in this

Table 13.1  Implications of Cole’s three questions for each stage of the research process Question Research stage Generation of hypothesis

Sampling

Operationalization

Analysis

Interpretation of findings

Who is included within What role does Where are the this category? inequality play? similarities? Is it attuned to Literature review May be exploratory diversity within attends to social rather than categories? and historical hypothesis testing contexts of to discover inequality. similarities. Focuses on neglected Category memberships Includes diverse groups. mark groups with groups connected unequal access by common to power and relationships resources. to social and institutional power. Develops measures If comparative, Views social categories from the differences are in terms of perspective of conceptualized as individual and the group being stemming from institutional studied. structural inequality practices rather (upstream) rather than primarily as than as primarily characteristics of individual-level individuals. differences. Attends to diversity Tests for both Interest is not limited within a group and similarities and to differences. may be conducted differences. separately for each group studied. No group’s findings Differences are Sensitivity to nuanced are interpreted interpreted in light variations to represent of groups’ structural across groups is a universal positions. maintained even or normative when similarities experience. are identified.

Source: Adapted from Cole, 2009, p. 172.

Applicability to MMR Critical reflection on marginalized and excluded groups.

All research methods should focus on neglected groups.

What procedures will safeguard the voices of the less powerful?

Intersectionalityinformed approaches to analyzing all the data. Avoid reproducing existing inequalities.

EXPLORING INTERLOCKING RELATIONSHIPS OF RACE, GENDER AND CLASS

study were more disadvantaged than the corresponding White young women and family life, religion and cultural identity were of central importance in some of their lives. However, the African-Caribbean young women in my study were less likely to smoke than their White peers, although African-Caribbean young women in the survey came from disadvantaged backgrounds. There is little research that has tried to theorize the interrelationship between cigarette smoking, gender, and race or ethnicity. Despite the growing literature on young women and cigarette smoking, to a large extent this literature has ignored the experience of Black and minority ethnic young women in general, and Black British young women specifically. Without a deep understanding of the cultural, historical, political, and social context within which a health behaviour is presented, it is difficult to make sense of the motivations and reasons for the particular health behaviour. Moreover, in developing intersectionality-informed research, the social and cultural context provides an understanding of the political privileges and disadvantages each group faces. In relation to Black and minority ethnic communities in the UK, there must be an attempt to understand the role that race and racism play in relation to the health status and health behaviour of minoritized groups. This chapter draws upon a tradition of developing appropriate research methodologies with Black and minority ethnic communities (Douglas, 1998a), and developing anti-racist research practices (Douglas, 1998b). I brought an intersectional perspective to this research, informed by wanting to understand the relationship between gender, race, class, and cigarette smoking behaviour. My study was designed to empirically address the gaps identified in the critical analysis of the literature. I adopted an intersectionality-informed mixed method approach utilizing quantitative and qualitative methodologies to address the research questions, as no one research method can examine both patterns and meanings of cigarette smoking effectively. Adopting an intersectional framework allowed the possibility of capturing the complexity of what may be perceived as one-dimensional categories such as race and ethnicity when exploring identity (Harper, 2011). Although the African, African-Caribbean and White young women in my research study occupy the same socioeconomic location, they make different choices about cigarette smoking. Using Cole’s three questions, I analyze my findings and the role that disadvantage plays in the lives of African, African-Caribbean and White young women, paying particular attention to the differences and similarities.

195

Study Background, Purpose and Research Questions The illustrative study aimed to explore both the reported patterns of smoking and the meaning that smoking has for young African-Caribbean women. As very little is known about the prevalence of smoking and the motivation for smoking among African-Caribbean young women, I adopted a mixed method approach, since no one research method could examine both patterns and meanings of cigarette smoking adequately. The study was guided by the following mixed methods research questions: 1 What are the patterns and influences on the cigarette smoking behaviour of African Caribbean young women in contemporary urban Britain? 2 What meaning does cigarette smoking have for African-Caribbean young women? 3 What do we learn from a mixed methods approach to researching the complex social phenomenon of cigarette smoking? In order to address these questions, I undertook a cross-sectional survey among Year 10 pupils, aged 14–15 years old in selected schools, and follow-up focus groups with African and African-Caribbean young women in Year 11, from two of these schools.

Black feminist epistemology My illustrative study design drew upon a Black feminist epistemology; by this, I mean a way of knowing which brings Black women to the centre of the analysis and examines Black women’s experiences in terms of race, class and gender (Mullings, 2000). Within an intersectional approach, gender does not exist as an independent category, but is always connected to race, class and ethnicity (Phoenix & Pattynama, 2006). Mirza (1997) argues that: Black British feminists reveal other ways of knowing that challenge the normative discourse. In our particular world shaped by processes of migration, nationalism, racism, popular culture and the media, Black British women, from multiple positions of difference, reveal the distorted ways in which dominant groups construct their assumptions. As Black women we see from the sidelines, from our space of unlocation, the unfolding project of domination. (p. 5)

Black feminism is concerned with power relations, racialized boundaries and the lived realities

196

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

of Black women. Intersectionality theory was developed in response to second-wave feminism, which privileged gender but did not give significance to race or ethnicity (Crenshaw, 1991). McCall (2005, p. 1771) argues that it “is the most important theoretical contribution that women’s studies, in conjunction with related fields, has made so far”. An intersectional approach considers the intersecting axes of oppression and discrimination that cut across each other and which may influence and change each other (Hankivsky & Christoffersen, 2008). A range of factors may influence health in different ways and may intersect differently at different stages of the life course (Hankivsky et  al., 2010; Hankivsky, 2012). In using a Black feminist framework for my study on young African-Caribbean women and cigarette smoking, I draw on intersectionality theory. In the 1980s, there were extended debates about what constituted feminist research. Several second-wave feminists challenged quantitative research methodologies as being inherently “male” (Harding, 1987; Haraway, 1991; Oakley, 1981; Stanley and Wise, 1983), arguing that such methods were positivist and reductionist, and perpetuated male privilege and male perspectives. Quantitative methods were further critiqued because of the exclusion of women as research respondents, and the notion that such research was supposedly objective, neutral and value free (Roberts, 1981). This early debate on positivist traditions versus interpretative traditions and reaction against quantitative methods led to the promotion of qualitative methods, which came to be viewed for a while as the orthodox feminist methodology (Hughes & Cohen, 2010). However, Reinharz and Davidman (1992), while reviewing feminist methods in social research, argued that feminism was a perspective, not a method, and depended on what use one made of the methods and how one went about one’s research.

Study Design, Procedures and Population The study used a mixed method, intersectionalityinformed research design which combined approaches from women’s studies and sociology of health to explore how race, class and gender influenced cigarette smoking. The study was conducted in three stages. In the first stage, selfcompletion questionnaires were used in classes with 15-year-old pupils in selected schools in a city in the West Midlands, England. The schools were selected based on the ethnic composition of pupils in the schools and the school data on free

school meals. This was used as a proxy to identify schools with disadvantaged pupils. The questionnaire was to enable a comparison of the influence of gender, ethnicity, and social class on cigarette smoking behaviour and perceptions of cigarette smoking across young people from different ethnic groups, by collecting data on self-reported patterns and influences on smoking behaviour from 700 young people. In the second stage, in-depth qualitative data was collected from first- and second-generation young African and African-Caribbean women, utilizing seven focus groups. The focus groups explored the factors that influenced smoking behaviour and the meanings that cigarette smoking had for this group. As I argued earlier, no research method is feminist as such. Wilkinson (1999) argues that focus groups have been underutilized by feminist researchers conducting social research. Wilkinson suggests that focus group methods address and promote the aims and goals of feminist researchers as they avoid an emphasis on isolated individual responses in surveys and one-to-one interviews by contextualizing the research. Also, they remove rigid hierarchies between the researchers and the researched. As there are more research participants than researchers, this redistributes the balance of power, and the research participants have the control to direct the focus of the discussion. This has, however, also been identified as a disadvantage of focus groups by some social researchers (Krueger, 1994). The advantage of focus groups is to allow the voices of the participants to be heard. Chiu and Knight (1999) argue that focus groups are a useful method for research with minority ethnic groups who are often marginalized in the research process. The focus group discussions allowed a nuanced debate about social, ethnic, racial and gender identity, and their relationship to cigarette smoking. Focus groups are a research method that enables the discussion of cultural values and beliefs (Bowling, 2002). While focus groups allow for the collection of data in a short space of time, one of the limitations of focus groups is that individuals in the group may feel influenced to project a view that correlates with others in the focus group. However, one of the key advantages of focus group interviews is that participants can challenge each other and allow the discussion to flow in the direction determined by the participants. This happened in my focus groups where individuals felt able to challenge each other and me. This was particularly true in relation to the discussions about identity and ethnicity, where the focus group participants debated whether they identified as “African”, “African-Caribbean” or “Black British”. Denny

EXPLORING INTERLOCKING RELATIONSHIPS OF RACE, GENDER AND CLASS

et  al. (2011) concluded that the use of focus groups when conducting research with minority ethnic communities allowed the data to be set in context, as participants not only discussed issues that were of particular significance to them, but also to their wider community. In the third stage, I used data triangulation to bring together the data from African, AfricanCaribbean and White young women in the survey with the findings from African-Caribbean young women in the focus groups. Through this process, I examined the extent to which and in what ways results from the two types of data converged, diverged and related to each other, to produce a more complete understanding. The study population for my survey was seven secondary schools in the West Midlands. Cigarette smoking has been linked to social disadvantage (Fergusson et  al., 2007) and I wanted to explore whether this was the case for the African, AfricanCaribbean and White young women, bearing in mind the complexities and tensions with using ethnic categorization. In this particular city, there were vast differences between schools. Some schools were predominantly Pakistani, others were predominantly African-Caribbean and some were exclusively White. The diverse representations in schools assisted with ethnic group quota sampling to try to obtain similar numbers from the five ethnic groups identified.

Stage 1: Survey Participants, Results and Discussion The final study sample was 701 young people. An attempt was made to obtain similar numbers of Bangladeshi, Pakistani, Indian, African-Caribbean and White young people, and equal numbers of young men and women. There are difficulties associated with undertaking school surveys that are dependent upon which students turn up on the day that the survey is being undertaken in the class. The final study population did not contain equal numbers of the five main ethnic groups but provided adequate numbers to enable a comparative study. The first stage of my research design involved a questionnaire administered by the classroom teachers. I decided to undertake a survey as there was very little information on the reported smoking patterns of African-Caribbean young women and other minority ethnic young people. The aim of the survey was to provide base-line data in order to compare smoking patterns in a cohort of young people in relation to gender, ethnicity and social class. The advantage of undertaking a survey is

197

that comparative information can be collected on a large number of people, identifying whether they smoke and when they started smoking. A crosssectional survey is a useful method for collecting standardized data on attitudes, knowledge and behaviour in relation to cigarette smoking. Survey research is a quantitative and positivist methodology. Although cross-sectional surveys can provide descriptive information about cigarette smoking and correlations or associations between different variables and factors, they cannot suggest cause and effect. A further limitation of a survey is that it is difficult to collect data on the meanings of cigarette smoking and perceptions of smoking. IBM SPSS Statistics 21 was used to analyze the data collected from the survey. A coding frame was developed and descriptive statistics were used to analyze reported cigarette smoking behaviour by ethnic group. Ethnicity was categorized using the 2001 census categories (ONS, 2001). However, on analysis, there were 34 different ethnic categories. As it would not be possible to undertake meaningful comparative analysis using 34 ethnic categories, for the purpose of further analysis, the ethnic categories were combined and recoded to eight main ethnic categories as follows: 1 White—White British, White Irish, White Other 2 Black—Black British, Black Caribbean/West Indian, Black Other 3 Black African 4 British Asian 5 Indian 6 Bangladeshi 7 Pakistani 8 Mixed Ethnicity—White and Black Caribbean, White and Black African, White and Asian 9 Other Mixed Ethnicity In relation to developing an intersectional framework, difficulties may be encountered with analyzing social surveys where gender, race and class are seen as independent variables, as an intersectionality framework sees these variables as interconnected, influencing and shaping each other. While some researchers may suggest that quantitative methods are antithetical to an intersectional analysis, Cole (2009) argues that “an intersectional analysis hinges on the conceptualization of race, gender, and other social categories, rather than the use (or avoidance) of particular methods” (p. 178). In my study, I explore the social categories used in the survey through the discussions in the focus groups, and I am able to explore the intersections between different social categories to provide deeper insights when examining cigarette smoking.

198

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Stage 2: Focus Groups Participants, Results and Discussion The participants for the focus groups were African and African-Caribbean young women recruited from Year 11 in the schools that participated in the survey, and were aged 16 and over. In the seven focus group discussions, information was collected using a flexible interview schedule. The focus group interviews explored attitudes to smoking, perceptions, beliefs, predispositions and experiences. This information was recorded and transcribed to ensure that the richness and texture of the data were not lost. I undertook seven focus groups. By that time, I had data saturation and the same main themes were occurring in each discussion. All the tapes from the focus groups with young AfricanCaribbean women were transcribed. I relistened to the tapes and edited the transcripts, as the person who transcribed the tapes did not always fully understand Caribbean dialect or Black British urban dialect (Regmi et al., 2010). The switching between English and Patois was very evident in the focus group discussions in my study, and the young women, both those born in the UK and the Caribbean, had an extensive command of Patois. While this presented difficulties for the English woman transcribing the recordings and required careful listening and relistening of the recordings for the analysis, it provided a rich and often humorous discussion. The young women’s speech specificities extended not only to the language that was used, but to the way it was used and, in some instances, the humor that was present in the focus groups was lost in translation and transcription (Regmi et al., 2010). Qualitative research with diverse groups needs to be cognizant of the different use of “English” language and the richness that this can bring to the research. The process of relistening to the tapes was helpful in enabling me to become immersed in the

data. The transcripts were analyzed using thematic analysis (Braun & Clarke, 2006). During this process of relistening to the tapes, I continued to conduct thematic analysis of the transcripts using constant comparison. The analysis of findings from focus groups can be time-consuming as, if participants are engaged, complex and messy data may be produced (Culley et al., 2007). Initially, I developed a coding frame from the data, with coding categories, and then developed subcategories.

Stage 3: Integration through Data triangulation Insights and Discussion My mixed method approach enabled the quantitative methodology and qualitative methodology to be combined to examine the same phenomenon of cigarette smoking in African-Caribbean young women. I adopted a convergent parallel design according to Creswell and Plano Clarke (2011). Thus, there was consecutive quantitative and qualitative data collection and analysis. This provided a more complete understanding of my research topic. I outline the stages of my research in Table 13.2. The data from African, African-Caribbean and White young women in the survey were triangulated with the findings from African-Caribbean young women in the focus groups. The two types of data provided a more complete understanding of the phenomenon of cigarette smoking in African-Caribbean young women. My mixedmethods approach enabled the quantitative methodology and a qualitative methodology to be combined to examine the complex social phenomenon of cigarette smoking in African-Caribbean young women. I adopted a convergent parallel design according to Creswell and Plano Clarke (2011). Thus, there was consecutive quantitative and qualitative data collection and analysis. The

Table 13.2  Cigarette smoking in African-Caribbean young women: flowchart of the basic procedures in implementing a convergent design Quantitative strand Qualitative strand Cross-sectional survey—numerical data Focus group data Analysis of quantitative data using IBM SPSS Statistics 21 Analysis of the qualitative data using thematic coding to to produce descriptive statistics. produce descriptive accounts, themes and subthemes. Strategies to merge the two sets of results based on themes generated: Triangulate the data from African, African-Caribbean and White young women in the survey with the findings from African-Caribbean young women in the focus groups. Discuss to what extent and in what ways results from the two types of data converge, diverge and relate to each other to produce a more complete understanding. Source: Author created.

EXPLORING INTERLOCKING RELATIONSHIPS OF RACE, GENDER AND CLASS

two data sources were then integrated and placed in conversation with each other in relation to specific themes. This provided a more complete understanding of my research topic. By using both methods—a cross-sectional survey and focus groups in a mixed-methods research design— the intersection of these two methods provided insights that I would not have gained using one method alone. Using an example from the focus groups with African and African-Caribbean young women, there was a widely held perception that White parents’ attitudes towards the cigarette smoking behaviour of their daughters was permissive. There was a view that White parents encouraged their children to smoke; when African-Caribbean young women visited the homes of their White friends, they were allowed to smoke cigarettes with them. This perception was not supported from the findings of the survey. When young women were asked in the survey what their family would do if they became aware that the respondent had started smoking, the responses were remarkably similar. What is striking is that there is little difference between the responses of the young White women and the young Black women here. This counters the views expressed by young Black women in the focus groups who thought that the families of young White women encouraged their children to smoke. These findings demonstrate the importance of a mixed methods research approach to tease out complexities and nuances in research.

CONTRIBUTIONS AND FUTURE DIRECTIONS FOR INTERSECTIONALITYINFORMED MIXED METHODS RESEARCH DESIGN FRAMEWORK This chapter makes two important contributions. First, the results discussion of the study example demonstrates the importance of mixed methods approaches for exploring health behaviours like cigarette smoking using an intersectionalityinformed mixed methods research design. Second, the detailed descriptions of the intersectionalityinformed mixed methods study example demonstrates the complexity of researching social differences, identity and power. Earlier research on cigarette smoking behaviour among young women has demonstrated the link between gender, social disadvantage and cigarette smoking, and concluded that the more socially disadvantaged young White women were, the more likely they were to smoke. Adopting an

199

intersectionality-informed approach to researching young women and cigarette-smoking by exploring the cigarette smoking behaviour of young AfricanCaribbean and African women in this study concluded that cigarette smoking was not associated with social disadvantage among young Black women, and that race, class and gender intersected to produce different outcomes in relation to their cigarette smoking behaviour. The mixed method, intersectionality-informed research approach was able to highlight this complexity. For a further discussion of integrating secondary data from ethnically and racially marginalized groups into mixed methods research, see also Chapter 25 (this volume). To take an intersectionality-informed approach, we need to investigate social groups within and across analytical categories so we can understand how those categories influence and change each other. At the same time, an intersectional lens allows an understanding of the political, historical and cultural factors that determine power in the context of the specific health behaviour. For a discussion of prioritizing cultural responsiveness in mixed methods research teams, see also Chapter 16 (this volume). The methodological and theoretical foundations of intersectionality can be formalized by building from empirical studies within particular disciplines (Cho et al., 2013, p. 792). There is limited research on young Black and minority communities and smoking, and comparative research on cigarette smoking is atheoretical. Research on White young people and White young women has linked cigarette smoking to social deprivation. However, when an intersectional approach is undertaken with African-Caribbean young women, this orthodoxy is disrupted. There is a need to develop theories that recognize the ways in which race ethnicity, class, religion and gender intersect with cigarette smoking in young AfricanCaribbean women. Rice and colleagues (2019) argue that for work to be meaningfully intersectional, it should address historical and contemporary social/ cultural forces through a political lens … [and] address why some levels and dimensions are the subject of focus, explaining what types of analysis might be facilitated and which might be limited as a result. (p. 416)

So, intersectionality requires us to consider context at the outset and throughout, and not add an intersectional interpretation as an afterthought. In relation to cigarette smoking, earlier research that examined socioeconomic factors and gender in relation to this health behaviour focused primarily

200

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

on cigarette smoking and White women. An intersectional approach to cigarette smoking enables an exploration of not just class and gender, but race, ethnicity, culture, religion and gender, and disrupts the orthodoxy that cigarette smoking is associated with material disadvantage. In examining cigarette smoking in African, AfricanCaribbean and White young women, this study, using an intersectionality-informed mixed methods design, demonstrates that even for young women in similarly materially disadvantaged circumstances, African and African-Caribbean young women are less likely to smoke because of the protective and intersecting nature of race, ethnicity, culture and religion. This intersectionality-informed mixed methods design expands and advances the field of mixed methods research. By adopting the three questions of Cole, mixed methods research is made more relevant to research studies with diverse populations. To others implementing this design approach, they should ask these three questions of their research. Health behaviour must be understood from the social and cultural context of people’s lives. Health promotion and public health research approaches should develop public health interventions in a more sustained manner from a social-determinants-of-health perspective—i.e., that socioeconomic and environmental factors influence health (Whitehead & Dahlgren, 2006). An intersectionality-informed approach is distinguished from a social determinants of health approach (Reid et  al., 2012) by the recognition that social categories are not simply additive (e.g., gender and race and class), but that something new is created and experienced at the intersection of one or more categories (Hankivsky et al., 2010). My study suggests that these categories intersect, and shape and influence each other, and cannot be viewed as individual, independent entities or individual variables. Hence, research methodologies must capture the interconnections between variables. Mixed methods are therefore possibly better suited to intersectionality-informed research. In relation to future directions for research, this study demonstrates that while the AfricanCaribbean young women in this sample were more disadvantaged than their White female peers, they were less likely to smoke. Caribbean culture, family life and religion were central to their lives and, to a large extent, protected many young women from cigarette smoking. As such, this research demonstrates findings based on predominantly one ethno-cultural group and does not necessarily translate to other groups, even if they live under similar material conditions. Thus, for future research on health behaviours and young people, an intersectionality-informed approach should be

adopted that not only considers race, gender and social class, but culture, religion and migration status.

WHAT TO READ NEXT Bowleg, L. (2012). The problem with the phrase women and minorities: Intersectionality – An important theoretical framework for public health. American Journal of Public Health, 102(7), 1267– 1273. https://doi.org/10.2105/AJPH.2012.300750

This article describes the history and central tenets of intersectionality, addresses some theoretical and methodological challenges, and highlights the benefits of intersectionality for public health theory, research and policy. Douglas, J. (2020). Developing Intersectionalityinformed Research Methodologies, Research Matters, Social Research Association. March 5. The-sra. org.uk. Available at: https://the-sra.org.uk/ (accessed October 15, 2022).

This article outlines the importance and benefits of an intersectionality-informed approach to research. Rice, C., Harrison, E., & Friedman, M. (2019). Doing justice to intersectionality in Research. Cultural Studies – Critical Methodologies, 19(6), 409–420. https://doi.org/10.1177/1532708619829779

This article traces the genealogy of intersectionality as theory and methodology to identify challenges in translating the concept into research methods.

REFERENCES Bowleg, L. (2008). When Black + lesbian + woman ≠ Black lesbian woman: The methodological challenges of qualitative and quantitative intersectionality research. Sex Roles, 59(5–6), 312–325. https:// doi.org/10.1007/s11199-008-9400-z Bowleg, L. (2012) The problem with the phrase women and minorities: Intersectionality – An important theoretical framework for public health. American Journal of Public Health, 102(7), 1267– 1273. https://doi.org/10.2105/AJPH.2012. 300750 Bowling, A. (2002). Research methods in health: Investigating health and health services. Open University Press. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology,

EXPLORING INTERLOCKING RELATIONSHIPS OF RACE, GENDER AND CLASS

3, 77–101. https://doi:10.1191/1478088706qp 063oa Chiu, L.-F., & Knight, D. (1999). How useful are focus groups for obtaining the views of minority groups. In R. Barbour & J. Kitzinger (Eds.), Developing focus group research: Politics, theory and practice (pp. 99–112). Sage. Cho, S., Crenshaw, K.W., & McCall, L. (2013). Towards a field of intersectionality studies: Theory, applications, and praxis. Signs, 38(4), 785–810. https://doi.org/10.1086/669608 Cole, E. R. (2009). Intersectionality and research in psychology, American Psychologist, 64, 170–180. https://doi.org/10.1037/a0014564 Collins, P. H. (1990). Black feminist thought: Knowledge, consciousness, and the politics of empowerment. HarperCollins. Crenshaw, K. (1989). Demarginalizing the intersections of race and sex: a black feminist critique of antidiscrimination doctrine, feminist theory and, antiracist politics. University of Chicago Legal Forum, 1989(1), 139–67. https://chicagounbound. uchicago.edu/uclf/vol1989/iss1/8 Crenshaw, K. (1991). Mapping the margins: Intersectionality, identity politics, and violence against women of color. Stanford Law Review, 43(6), 1241–1299. https://doi.org/10.2307/1229039 Creswell, J. & Plano Clark, V. (2011). Designing and conducting mixed methods research. Sage. Culley, L., Hudson, L., & Rapport, F. (2007). Using focus groups with minority ethnic communities: Researching infertility in British South Asian communities. Qualitative Health Research, 17, 102– 112. https://doi.org/10.1177/1049732306296506 Davis, A. (1981) Women, race and class. The Women’s Press. Denis, A. (2008). Review essay: Intersectional analysis: a contribution of feminism to sociology. International Sociology, 23(5), 677–694. https://doi. org/10.1177/0268580908094468 Denny, E., Culley, L., Papadopoulos, I., & Apenteng, P. (2011). From womanhood to endometriosis: Findings from focus groups with women from different ethnic groups. Diversity in Health and Care, 8, 167–80. ISSN: 2049–5471 Douglas, J. (1998a). Developing appropriate research methodologies with black and minority ethnic communities. Part 1: reflections on the research process. Health Education Journal, 57(4), 329–338. https://doi.org/10.1177/001789699805700405 Douglas, J. (1998b). Working with an anti-racist perspective. In J. Douglas (Ed.), Health and social care research: Developing appropriate research methodologies with black and minority ethnic communities. Conference Report. SHARP/Sandwell Health Promotion Unit/ Sandwell Health Authority. Douglas, J. (2014). African-Caribbean young women in the UK and cigarette smoking. Unpublished PhD thesis, University of York.

201

Douglas, J. (2018). The politics of black women’s health in the UK – intersections of ‘race’, class and gender in policy, practice and research. In N. Alexander-Floyd & J. Jordan-Zachery (Eds.), Black women and politics: demanding citizenship, challenging power, and seeking justice. (pp. 4–68). SUNY Press. Douglas, J. (2020). Developing intersectionalityinformed research methodologies, Research Matters, Social Research Association. March 2020: 5 The-sra.org.uk. Available at: https://the-sra.org.uk (accessed October 15, 2022). Douglas, J., & Watson, N. (2013). Editorial: Resistance, resilience and renewal: The health and well-being of black women in the Atlantic diaspora – developing an intersectional approach. Critical Public Health, 23(1), 1–5. https://doi.org/10.1080/ 09581596.2013.760724 Fergusson, D.M., Horwood, L.J., Boden, J.M., & Jenkin, G. (2007). Childhood social disadvantage and smoking in adulthood: Results of a 25-year longitudinal study. Addiction, 102(3), 475–482. https://doi:10.1111/j.1360-0443.2006.01729 Grace, D. (2014). Intersectionality-informed mixed method research: A Primer. Institute for intersectionality Research and Policy, Simon Fraser University. Hankivsky, O. (Ed). (2012). An intersectionality-based policy analysis framework. Institute for Intersectionality Research and Policy, Simon Fraser University. Hankivsky, O. and Christoffersen, A. (2008). Intersectionality and the determinants of health: A Canadian perspective. Critical Public Health, 18(3): 271–283. Hankivsky, O., Reid, C., Cormier, R., Varcoe, C., Clark, N., Benoit, C. and Brotman. S. (2010). Exploring the promises of intersectionality for advancing women’s health research. International Journal for Equity in Health, 9: 5. Hankinsky, O., & Grace, D. (2015). Understanding and emphasizing difference and intersectionality in multimethod and mixed methods research. In S. Nagy Hess-Biber & R. B. Johnson. The Oxford handbook of multimethod and mixed methods research inquiry (pp. 110–127). Oxford University Press. Hankivsky, O., & Jordan- Zachary, J. S. (2019). Introduction: Bringing intersectionality to public policy. In O. Hankivsky & J. S. Jordan-Zachary (Eds), The Palgrave handbook of intersectionality in public policy (pp. 1–28). Palgrave Macmillan. Haraway, D. (1991). Situated knowledges: The science question in feminism and the privilege of partial perspective. In D. Haraway (Ed.), Simians, cyborgs, and women. The reinvention of nature (pp. 183–202). Free Association Books. Harding, S. (1987). Introduction: Is there a feminist method? In S. Harding (Ed.), Feminism and methodology (pp. 1–14). Indiana University Press.

202

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Harper, C. (2011). Identity, intersectionality, and mixed-methods approaches. New Directions for Institutional Research, 151, 103–115. https://doi. org/10.1002/ir.402 Hughes, C., & Cohen, R. (2010). Feminists really do count: The complexity of feminist methodologies. International Journal of Social Research Methodology, 13(3), 189–196. https://doi.org/10.1080/ 13645579.2010.482249 Krueger, R.A. (1994). Focus groups: A practical guide for applied research. Sage. Lewis, G. (2013). Unsafe travel: Experiencing intersectionality and feminist displacements. Signs, 38(4), 869–892. https://doi.org/10.1086/669609 McCall, L. (2005). The complexity of intersectionality. Signs, 30(3), 1771–1800. https://doi.org/10.1086/ 426800 Mirza, H. (1997). Black British feminism: A reader. Routledge. Mirza, H. (2009) Race, Gender and Educational Desire: Why Black Women Succeed and Fail. Routledge. Mullings, L. (2000). African-American women making themselves: Notes on the role of black feminist research. Souls: A Critical Journal of Black Politics, Culture, and Society, 2(4), 18–29. https:// doi.org/10.1080/10999940009362233 Oakley, A. (1981). Interviewing women: A contradiction in terms. In H. Roberts (Ed.), Doing feminist research (pp. 30–61). Routledge & Kegan Paul. ONS (2001). 2001 census data – Office for national statistics. www.ons.gov.uk/census/2011censusdata/ 2001censusdata (accessed October 15, 2022). Onwuegbuzie, A. J., Abrams, S. S., Forzani, E., & Natesan Batley, P. (in press). The many SIDES of critical dialectical pluralism: A meta-philosophy— comprising a research philosophy, educational philosophy, and life philosophy—for addressing social justice, inclusion, diversity, and equity, and social responsibility. International Journal of Multiple Research Approaches.

Phoenix, A., & Pattynama, P. (2006). Special issue on intersectionality. European Journal of Women’s Studies, 13(3). https://doi.org/10.1177/135050 606806065751 Regmi, K., Naidoo, J., & Pilkington, P. (2010). Understanding the process of translation and transliteration. International Journal of Qualitative Methods, 9(1), 16–26. https://doi.org/10.1177/ 160940691000900103 Reid, C., Pederson, A. and Dupere, S. (2012). Addressing Diversity and Inequities in Health Promotion: The Implications of Intersectional Theory. In I. Rootman, S. Dupere, A. Pederson, and M. O’Neill (eds.). Health Promotion in Canada: Critical Perspectives on Practice. Canadian Scholar’s Press Inc., 54–56. Reinharz, S., & Davidman, L. (1992). Feminist methods in social research. Oxford University Press. Rice, C., Harrison, E., & Friedman, M. (2019). Doing justice to intersectionality in research. Cultural Studies – Critical Methodologies, 19(6), 409–420. https://doi.org/10.1177/1532708619829779 Roberts, H. (1981). Doing feminist research. Routledge & Kegan Paul. Shannon-Baker, P. (2021). Centering race in mixed and multi-method research on implicit bias: A systematic review. International Journal of Multiple Research Approaches, 13(1), 55–73. https:// doi.org/10.29034/ijmra.v13n1a3 Stanley, L., & Wise, S. (1983). Breaking out: Feminist consciousness and feminist research. Routledge & Kegan Paul. Whitehead, M. and Dahlgren, G. (2006). Concepts and principles for tackling social inequities in health: Levelling up Part 1. World Health Organization: Studies on social and economic determinants of population health 2,460–474. Wilkinson, S. (1999). How useful are focus groups in feminist research? In R. Barbour and J. Kitzinger (eds.). Developing Focus Group Research – Politics, Theory and Practice. Sage, 64–78.

14 Indigenous Cultural Values Instrument Development: Using Mixed Methods Research M e h d i Ta g h i p o o r r e y n e h

INTRODUCTION Cultural values are the intangible beliefs that exist in the minds of a group of people (Hofstede, 1980). These values are formed due to human interaction with their environment to achieve the best way to live together (Schwartz, 1999). The intangibility of these cultural values requires the use of specific methods to identify and measure them. On one hand, researchers are aware of how quantitative methods can increase the reliability of cultural values instruments. On the other hand, they also know that cultural values are so complex and diverse that they can be best understood through qualitative methods. As such, a successful assessment of cultural values must rely on some combination of the two methods to benefit from the strength of each method. This chapter will provide an illustrative example of how mixed methods research can be used as an effective method in recognizing and measuring these cultural values. In this research, first, the author reviews the concept of cultural values assessment approaches, which involves using one or more analysis types associated with one tradition (e.g., qualitative analysis) to analyze data associated with a different tradition (e.g., mixed methods analysis). Second, the author discusses the context of the

research and the need for the development of the instruments at indigenous level. Third, the author conducts a three-round Delphi study to integrate open-ended and free-listing data, and to generate a list of items. The items were subjected to a field test to ensure reliability. A correspondence analysis of scale was employed to examine the validity of the instrument. Finally, the author outlines how qualitative and quantitative approaches can be combined to enhance instrument reliability and validity at different stages of the development process. The results indicated that the mixed methods approach allowed the development of instruments based on the inclusion of indigenous participants and consultation with local experts. It also demonstrated the value of Delphi groups as part of a mixed methods research design in advancing the validity and reliability of a quantitative cultural values instrument.

VALUES, CULTURAL VALUES AND CULTURE Although the terms “values”, “cultural values” and “culture” are often used interchangeably, each term is actually a distinct piece of the larger picture. Values are defined as desirable goals that guide people to select and evaluate actions, policies and

204

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

events (Schwartz, 1999). Values operate at the individual and group levels. At the individual level, values express broad desirable goals that guide and affect the way individuals interpret and perceive events and situations (Schwartz, 1999). At the group level, values are shared conceptions of what is good, right and desirable in solution groups (Hofstede, 1980; Schwartz, 1999). Cultural values are the group values that members of a society share and develop in response to existential challenges (Hofstede, 1980; Schwartz, 1999). Cultural values therefore play a crucial role in the way that social institutions function. Culture is defined as all of a group’s guiding values, rituals, heroes and symbols taken together as one big whole (Hofstede, 1980).

LIMITATIONS WITH CULTURAL VALUES ASSESSMENTS A long-standing challenge in studying the relationship between values and culture has proposed two dominant ways to view culture. One way is to view culture from others’ lenses and to compare individuals’ values across cultures. Generally, the goal of such a view is to compare cultural values differences across cultures. Typically, instrument developers conceptualize items to develop both theory-based (Inglehart, 2003; Schwartz, 1999) and non-theory-based (Hofstede, 1980; Rokeach, 1973) instruments to measure shared values (Triandis, 1996) and compare them across national levels. To do so, researchers relied on quantitative approaches for comparing cultural values scores and to test the hypothesized relationships among cultures as an independent variable, along with dependent variables. The main strength of such an approach is the reliability of the results, which are a product of the large empirical database in this area (Cheung et al., 2011). Attempts to use cultural values instruments developed by this approach have encountered several challenges, including substantive and methodological challenges. One of the main substantive challenges involves the implied emphasis on Western values, which may not be the area of concern for other cultures (McCrae et  al., 2005; Muthukrishna et al., 2020). What may be missing from imposed Western measures are indigenous values that are salient and silent in a local culture. In other words, it may lead researchers to overestimate values that are not important and to underrate values that are important for an indigenous culture (Cheung et al., 2011).

The methodological challenges relate to use of equivalence test for evaluating universality. The equivalence bias can arise from three sources: method, construct and items (Cheung et al., 2011; Ho, 1996). One empirical example of construct bias can be found in Ho’s (1996) study on filial piety. The Chinese concept, which refers to the expectation that children should have to take care of their elderly parents, is broader than the corresponding Western conception, which concentrates more on respect and love toward parents. In terms of items, mostly prior studies used the Likert scale to assess the differences across cultures. The question is whether the level of agreement or importance between cultures is the same. Studies have found differences in responding to survey questions between different racial/ethnic groups (Hui & Triandis, 1989). Their finding indicated that high collectivists are more likely to avoid extreme responses than low collectivists (Triandis, 1996). Hoy (1993) has referred to this “aversion to the spotlight” as “cultural shyness”. The other way is to look at values in specific cultural contexts. This approach promotes a comprehensive understanding of the culture being studied through a “thick description” (Geertz, 1973). This approach concentrates on qualitative methods of cultural assessment (Lenartowicz & Roth, 1999) and primarily follows anthropologists’ points of view that cultures are so diverse and complex that they cannot be measured, but merely described and observed (Haviland, 1990). Such an approach provides descriptive data that delineate cultural characteristics. It began as a reaction to the dominance of Western instruments, which did not provide comprehensive and sensitive enough instruments for understanding human behaviour in non-Western contexts (Kim et  al., 2006). Adair (2006) considered the cultural differences and distance from the American context as criteria for the indigenization of the instruments and disciplines. The sensitivity of the instruments that have been developed by these early indigenization movements has been considered as the main strength of the qualitative approach. Despite the sensitivity, the reliability of the results produce from the qualitative approach is still an issue. One of the limitations of this approach is the tendency of local researchers to emphasize the value uniqueness of the culture being studied (Cheung et  al., 2011). This may underrate the potential of the universal studies that undertake to explain indigenous values (Cheung et al., 2011). Church (2001) critiqued the fact that in attempting to distinguish universal and indigenous approach differences, many indigenous measures identified values that could also be subsumed under the universal instruments.

INDIGENOUS CULTURAL VALUES INSTRUMENT DEVELOPMENT

To strike a balance between the reliability and the sensitivity of the cultural values instrument, researchers must rely on some combination of the qualitative and quantitative methods (see Chapter 20, this volume). Combining these methods is necessary for advancing our comprehension of cultural values because it allows for the development of a theory-based understanding of values instead of just as an instrument. It also allows the determination of top values, and helps researchers to identify the full domain of the values in their studies to ensure comprehensive coverage of cultural values and their relevance to the indigenous context. Although it seems useful to combine methods to study cultural values, few practical examples exist to mix these methods together. One reason may arise from the traditional philosophical assumptions that quantitative and qualitative methods cannot be combined and used together. Another issue may be the scarcity of good examples that researchers can refer to. However, the purpose of this chapter is to provide an example that can show the usefulness of the mixed methods for the development of measuring instruments and filling the existing gap.

ORIENTATION TO MALAYSIAN CULTURAL CONTEXT Asia, with its wide spectrum of people from different cultural backgrounds, has always been and continues to be an interesting place for advertisers. Among the Asian countries, Malaysia has a remarkable cultural diversity and provides a fertile ground for study on various aspects of culture. The main ethnic groups within Malaysia comprise Malays, Chinese and Indians, with many other ethnic groups represented in smaller numbers (Asma, 1992). Most of them still practice their own local customs and laws. In Malaysia, Malays speak in Bahasa, practice their own customs, and follow Islam as their religion (Asma, 1992). Most of the Chinese speak Mandarin in their daily lives, with their own dialects such as Cantonese (Lee & Tan, 2000). They are followers of various religions and practices, such as Buddhism and Christianity (Asma, 1992; Purcell, 1965). Indians in Malaysia live mainly in West Malaysia and many still follow their customs from their country of origin, India. Despite clear differences in symbols, heroes and rituals, most of the previous studies in this context could not find significant differences in cultural values among these ethnic

205

groups (Fontaine & Richardson, 2005; Lim, 2001; Terpstra-Tong, 2014). Most Malaysian scholars agree that the current Western cultural instruments are not sensitive enough to explain and compare cultural values within and between ethnic groups in Malaysia (Asma, 1992, 2001; Terpstra-Tong, 2014). There is a need for more investigation of ethnic values in Malaysia to develop sensitive instruments responsive to Malaysians (Asma, 1992, 2001; Lim, 1998). In this study, focus is given on the largest ethnic group in Malaysia: the Malay. Generally, a Malay as defined in article 160(2) of the Malaysian constitution is a person who professes Islam, habitually speaks the Malay language, and conforms to Malay customs.

ILLUSTRATIVE EXAMPLE OF MIXED METHODS RESEARCH DESIGN FOR INSTRUMENT DEVELOPMENT An exploratory sequential mixed methods design, this section is composed of two phases—mixedmethods Delphi study and field-test phase. The primary reason for applying Delphi for this study is that it has been successfully used for achieving consensus on complex matters where precise and accurate information is not available (e.g., Addison, 2003; Briedenhann & Wicknes, 2007; Hayes, 2007; Linstone & Turroff, 2002). In fact, the Delphi technique is a mixed method in nature that utilizes both quantitative and qualitative strategies (Hasson et al., 2000). The Delphi study also allows for greater participation and inclusion from experts in indigenous cultures who are often left out by traditional instrument developers due to the quantitative nature of their research methods. It was useful for attempting to confirm, crossvalidate, and integrate research findings. Once the instrument was developed, it was necessarily subjected to a field test. This field test optimally would represent a sequential exploratory design to assess the appropriateness of each item. The main steps for the mixed methods process are depicted in Figure 14.1.

Delphi Study The Delphi study began with the critical qualitative opening round wherein the experts were asked to provide responses to open-ended questions concerning the Malay culture. After generating the data input, the emphasis switched to

206

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

quantitative analysis and the collection of data to achieve consensus of the experts’ opinions on the emerged values and items. At the same time, the experts were asked to respond to other openended questions for clustering proposes. The main steps for the mixed Delphi process and study data-collection procedure are depicted in Figure 14.2. The necessary steps for this technique will be described as follows.

Panel of Experts In this study, the expertise of the panel member was defined based on their experience and knowledge toward the Malay culture. The experts’ level of knowledge was assessed based on their scientific profile and publication. The experts for this study were a total of seven experts selected by purposive sampling. The sample of qualified experts was taken from local higher learning institution such as University of Malaya (UM), Universiti Putra Malaysia (UPM), Universiti Utara Malaysia (UUM) and Universiti Malaysia Sarawak (UNIMAS), and from cultural institutions. All panel members had a PhD degree relevant to Malay culture and were actively involved in the areas of Anthropology, History and Communications. A panel of 5–10 members was assumed to be enough for an exploratory study of a homogeneous population (Garson, 2014; Martino 1972). All panel members had a PhD degree relevant to Malay culture and were actively involved in Anthropology, History and Communication.

Number of Rounds Three rounds were found to be sufficient to yield consensus of the panelists. This three-round Delphi process involved a series of questionnaires in which a questionnaire was subsequently formulated and built from respondent feedback, and comments to the preceding questionnaire. The number of rounds depended upon a reasonable level of consensus and sufficient information (Delbecq et al., 1975; Ludwig, 1997; Sobaih et al, 2012).

First Round The purpose of round one was to generate a comprehensive list of values relating to the Malay

culture. The round used open-ended questions to glean as much information as possible (Taylor & Judd, 1989) and in a bid to reflect the exploratory nature of the study (Miler, 2001). Experts were asked to: (1) describe their ethnic groups; (2) underlie specific characteristics of Malays; and (3) indicate up to five (or ten) basic and fundamental values of Malays. Two first questions provided background information about Malay values. The third question asked participants to name the basic values among Malay people. By using inductive analysis, repeated words from the initial list of values allowed us to indicate Malay cultural orientations. The idea is that a value that appears frequently in all the documents is probably of more importance than a value that appears infrequently in the documents. The results of the first round of the study were used in the second round.

First Round Result The panelists were offered a list of values that they perceived as the most relevant to the Malay culture. Of the 22 suggestions, the five most frequently mentioned were politeness, having respect for elders, being accommodating, indirectness and collectivism. The most expressed value was politeness, mentioned by experts in 13.63 per cent of their responses. Other common values, such as having respect for elders and being accommodating, were seen as essential in 11.36 per cent of all responses. Indirectness and collectivism also came under close attention by the experts. The remaining values were scattered among the literal response, ranging from 6.81 per cent to 2.27 per cent in a number of occurrences. To assess the face validity, the researchers looked for values that were mentioned by at least two experts in order to proceed to the next rounds (Bearden et al., 1989).

Second Round: Qualitative Data Collection Leading to Categorizing Value Dimensions The purpose of the second round was to achieve experts’ consensus on the values that emerged from the first round, and to categorize values into the distinct dimensions by searching for crossthemes. Although the first-round instrumentation presented open-ended statements to generate depth and breadth of input (divergence), the

INDIGENOUS CULTURAL VALUES INSTRUMENT DEVELOPMENT

207

Figure 14.1  Procedural diagram of using a mixed method to create an indigenous cultural values instrument second round initiated the process of convergence. This round included two sections. In the first section, the panel members provided data through close-ended questions on a Likerttype scale. This scale made it possible to score the list of values from 1 (not important at all) to 5 (very important). A total score for each value emerged from the statistical analysis. Descriptive analysis was used to calculate the median for the total score. This was considered as the cut-off

point to achieve consensus. The total score was defined as a summation of the experts’ importance rating toward each value. The median is the most accurate statistical approach to show group views (Martino, 1972. This method is in line with the average majority consensus cut-off rate that has been sporadically used in Delphi studies (Heiko, 2012). Only those responses receiving a median score higher than the cut-off point were used in the next round. In the second section, experts were

208

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Third Round (Highlights)

 Items were generated  First pilot study was conducted to improve clarity of the items.  Second pilot study was implemented to avoid any double-barrelled statements.  More than 80% agreement on 6-point Likert scale on the top 3 measures was considered consensus.  For final evaluation, experts were asked to evaluate the degree of the relevance of the items on the instrument.

• • •

•

Second Round (Highlights)

 Experts rated their agreement level on 5point Likert scale toward the synthesized results (q1 and q2).  Experts submitted additional comments on the synthesized results  Experts rated the importance level of each value in the lists on 5-point Likert scale  Values having an importance level below the cut-off point (median for total score=21) have not reached consensus and were excluded in the next round.  Experts defined consensus values for clustering.  Values and its dimensions were used to develop an instrument in the third round

Post-Delphi process Final report Request to comments on the instruments Testing instrument among the population Pre-Delphi Process Administrative supports

(Letters of invitation, Reminding letters, Human subject issues)

• • •

•

•

• •

Define the research problem Develop research design Selection of expert panellists (based on selection criteria) Design and assessment of the First Round questions

First Round (Highlights)

Experts were asked to describe Malay ethnic groups and indicate basic and fundamental values among them Results were synthesized based on emerging themes for use in the second round (q1&q2) Lists of values were developed for evaluation by experts in the second round (q3)

Figure 14.2  The Delphi process asked to define the most repeated values from their point of view to reduce overlap across the themes. This step led to the creation of a hierarchy of categories.

Second Round Results As presented in Table 14.1, the values frequently mentioned by the experts rated a higher score. The

findings confirmed our assumptions in the first round. The median cut-off point for this step was found to be 21. In other words, if the total score for each value was greater than 21, consensus is reached. Of the eleven values, six, including politeness, respect to elders, being accommodating, collectivism, indirectness and hierarchy, are consistent with the results of the first round and reached consensus, whereas five values did not. These consensus values proceeded to the next step, clustering.

209

INDIGENOUS CULTURAL VALUES INSTRUMENT DEVELOPMENT

Table 14.1  Importance rate and expert consensus Importance rate E*1

E2

E3

E6

E7

Politeness Respect to elders Being accommodating Indirectness Collectivism Tolerance Helpfulness

5 5 5 5 5 5 4

5 4 4 3 5 4 3

5 5 4 4 4 4 4

5 5 5 5 4 4 5

3 3 3 5 4 3 4

23 22 21 22 22 20 20

Achieved Achieved Achieved Achieved Achieved Not achieved Not achieved

Relationship Hierarchy Modesty Cultural practice

4 4 4 4

4 4 4 3

4 4 4 4

4 5 4 5

3 4 3 4

19 21 19 20

Not achieved Achieved Not achieved Not achieved

Values

Total Score

Consensus

*E indicates expert.

The most common themes that emerged from section one were (a) politeness, (b) respect to elders, (c) being accommodating, (d) indirectness, (e) collectivism, and (c) hierarchy. The experts were asked to define these values to reduce overlap across the themes. A summary of the experts’ responses is illustrated in Table 14.2. The following is a discussion of the data by themes.

Third round: Quantitative Data Collection Leading to Achieve Consensus Toward Survey Items Generated from Qualitative Data Collection The purpose of round three was to generate items based upon the cultural values that emerged from the previous rounds. This round was composed of two sections. The first was to develop a set of items related to Malay values, and the second was to ensure that items developed were viable enough to present these values. In the first section, respondents were asked to compare themselves with statements in each item. The levels of similarity with the items were arranged from 1 (not like me at all) to 6 (very much like me) on a Likerttype scale. The second section set out to obtain experts’ agreement on the most relevant items that present Malay values. In this section, experts were asked to evaluate the degree of the relevance of the items on the instrument, asking the question, “How much is the person in the description like a Malay?” with the choice of answer being: 6 (very

much like a Malay), 5 (like a Malay), 4 (somewhat like a Malay), 3 (a little like a Malay, 2 (not like a Malay), and 1 (not like a Malay at all). Consensus was assessed by five experts who participated in the two previous rounds. For this section, more than 80 per cent agreement on a six-point Likert scale on the top three measures (very much like a Malay, like a Malay, and somewhat like a Malay) was considered the consensus. Items with lower ratings were removed from the instrument. The same approach has been used by Putnam et al. (1995). We decided to be conservative and use 80 per cent as the cut-off point for this final evaluation to identify the instrument that would be acceptable for use in future research. Following this, after providing consensus statements, results can then be applied to the quantitative stage.

Third-Round Results In sum, 16 items were developed to measure the various aspects of the Malay culture. In addition, three items were adapted from the hierarchy dimension of Schwartz’s (2003) scale, as they represented the findings very well. After conducting two pilot studies, seven items were modified to improve clarity, and one item was eliminated because of a median less than three. Finally, a total of 17 items had reached consensus upon completion of round three, as shown in Table 14.3. This represents 94  per cent of the total number of the items. Of these, five items reached the minimum cut-off rate at 80 per cent and 12 reached it at 100 per cent. The final instrument is shown in Table 14.4.

210

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 14.2  Malay values and their dimensions Dimensions

Values

Experts’ thoughts

Politeness

Respect to elders Indirectness Being tolerant

ü “Politeness means being patient in dealing with others.” (Expert 7) ü “Lending them (others) your ears, elder first.” (Expert 6) ü “Respect and try to ensure not to offend or hurt other feelings. As much as possible, he/she tries to understand other people or other ethnic and try to make them happy.” (Expert 3)

Collectivism

Group preferences Majority consensus Duty Reciprocation of favor

ü “Collectivist means being comfortable working in groups rather than alone. Teamwork is important.” (Expert 1) ü “Teamwork is important.” (Expert 1) ü “Collectivism means seeking for the majority consensus.” (Expert 6 and Expert 7) ü “Collective values refers to values that are held together or practised together by the community. This value has been practised for a long time, such as helpfulness, gotong-royong [working together for the community] and concern for neighbors.” (Expert 3)

Being accommodating

Appreciate differences Being flexible

ü “Being accommodating is being sensitive to the feelings and nuances of the others.” (Expert 1) ü “Willingness to accept other people.” (Expert 7) ü “The Malays are willing to accept other ethnic groups or others in accordance with their willingness to compromise. An accommodating attitude caused others to respect them.” (Expert 3) ü “Accommodating attitude refers to the willingness to accept others without prejudice and bias.” (Expert 3) ü “Malays are very open minded.” (Expert 2). ü “Being accommodating is more about flexibility.” (Expert 6)

Hierarchy

Family relationship Power Wealth

ü “A cultural dimension which states that in some cultures emphasis is placed on rank, status, and other ascribed attributes over equality issues.” (Expert 1) ü “Referring to the social status in the community, age, family relationship.” (Expert 3) ü “Accepting the position of power.” (Expert 6)

Field-Test Study: Quantitative Data Collection The purpose of this study was to ensure that the items developed in the Delphi study were viable and reliable. To this end, a purposive sample of 352 Malays with various backgrounds from across Malaysia was obtained. All items were assessed on a seven-point Likert-type scale from 1 (not like me at all) to 7 (totally like me). The summary of respondents’ demographic profiles is shown in Table 14.5.

Comparison Between Experts and Sample Results The result of the field test is presented in Table 14.6. The finding demonstrated that the pattern of results is like the experts’ final evaluation results.

The items related to politeness received the highest mean score among the Malay respondents. Similar results were found for being accommodating, collectivism, and hierarchy items. As expected, hierarchy items received the lowest mean score among respondents. Therefore, the findings from this study led us to conclude that the items developed in the Delphi study were reliable in representing Malay values.

Correspondence Analyses To demonstrate the sensitivity of the cultural value instrument, we used correspondence analysis of the aggregated responses (Maltseva, 2016). The analysis allows the visualizing mutual positioning of cultural values relative to each other, with respect to demographic information in a two-dimensional space. The output is shown in Figure 14.3.

Proper behavior Tolerant Indirectness to others Indirectness to elders Respect for elders Sensitive Flexible Appreciate differences Majority consensus Teamworking Duty Advice Community Family Wealth Influential Obedient Power distance

Items’ A little like Malay

Not like Malay

Not like Malay at all

2 3

2

1 2 2 1 1 2

1

2 2 3

3 5 3 2 2 4 2 4 1 1 1 1 2 2 2

2

1 2 1 3 1 1 2 1 1 2 1 1 2 1 3

1 1 1 1

1

100% 100% 100% 100% 100% 100% 100% 100% 100% 80% 80% 80% 80% 100% 80% 20% 100% 100%

Top three measures

Somewhat like Malay

Very much like Malay

Like Malay

Number of opinions

Number of answers

Table 14.3  The experts’ final evaluation results

Achieved Achieved Achieved Achieved Achieved Achieved Achieved Achieved Achieved Achieved Achieved Achieved Achieved Achieved Achieved Not Achieved Achieved

Consensus

Total results

INDIGENOUS CULTURAL VALUES INSTRUMENT DEVELOPMENT 211

212

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 14.4  Final Malay value items Items P1 P2 P3 P4 P5 A1 A2 A3 C1 C2 C3 C4 C5 H1 H2 H3 H4

It is important to her/him to be polite to other people all the time. She/he tries to ensure not to offend or hurt the feelings of others. (Proper Behavior) It is important to her/him to be tolerant in dealing with her/his elders. She/he believes she should always listen to those who are senior in age, even if she/he does not like what they say and do. (Being tolerant) She/he tries to be indirect when communicating with others. It is important for her/him to be sensitive about the face value of her/his hearers. (Indirectness to others) It is important for her/him to treat her/his elders with courtesy. She/he tries to express her/his real feelings indirectly when she/he is not satisfied with them. (Indirectness to elders) It is important to her/him to respect elders. She/he believes she/he should always show respect to her/his parents and to older people. (Respect to elders) It is important to her/him to be sensitive to how others might feel when communicating. She/he tries not to disturb or irritate others. (Being Sensitive) It is important to her/him to be flexible in behaving with others. How she/he behaves with others depends on who she/he is with, where she/he is, and both. (Being flexible) It is important to her/him to understand that human beings are not the same. She/he thinks she should always appreciate differences toward others and those from other races. (Appreciate differences) It is important to her/him to be socially accepted. She/he wants to avoid doing anything that people would say is wrong. (Majority consensus) Teamwork is very important to her/him. She/he is more comfortable working in a group rather than alone. (Teamwork) It is important to her/him to fulfill his duty toward his group members. She/he believes she/he should help them, within her/his means, when they are in trouble. (Duty) It is important for her/him to turn to others close to her for help in making decisions. Before making decisions she/he always consults others. (Advice) It is important for her/him to be concerned about her/his community. She/he would like to participate in practice such as gotong-royong to help her/his community when they in trouble (Community concern) It is important to her/him to listen to her family members. She/he tends to follow their advice without asking any questions. (Family relationship) It is important to her/him to be rich. She/he wants to have a lot of money and expensive things. (Wealth) It is important to her/him to be obedient. She/he finds it hard to disagree with someone in higher position than her. (Obedient) It is important to him/her to accept power distance in society. He/she thinks a person’s social status reflects his/her place in society. (Power)

Note: P = Politeness A = Accommodating C = Collectivism H = Hierarchy

Values data were split, based on the median score for each item, into high- and low-value orientation to be able to attribute and emphasize the differences originating from the respondents’ demographic profiles. The dispersion of demographic characteristics on the graph and their unequal distance from highand low-value orientations suggest that Malays from different demographic profiles possess different cultural values. The equidistant position

of genders from the high- and low-value orientations indicates the absence of strong differences in value orientation between Malay males and females. However, females have a slightly higher sense of belongingness to their parents’ culture. In contrast, the wide distribution of the respondents’ geographic profiles around the graph indicates the importance of this factor in shaping Malay value orientations. Furthermore, Malay income level appears to be a more influential factor than gender

213

INDIGENOUS CULTURAL VALUES INSTRUMENT DEVELOPMENT

Table 14.5  Respondents’ demographic profile Variable

Item

Frequency

Percentage

Gender

Male Female 18–28 29–39 40–50 51–60 Johor Kedah Kelantan Melaka Pahang Penang Perak Petaling Sabah Sarawak Selangor Trengganu Below 1,000 1,000–2,000 2,001–3,000 3,001–4,000 4,001–5,000 Above 5,000

149 203 104 186 50 12 45 15 29 93 32 4 15 1 12 49 48 9 71 89 77 36 28 51

42.3 57.7 29.5 52.8 14.2 3.4 12.8 4.3 8.2 26.4 9.1 1.1 4.3 0.3 3.4 13.9 13.6 2.6 20.2 25.3 21.9 10.2 8.0 14.5

Age

Place

Income level

Table 14.6  Results of the field test Items 1

Politeness

2

Accommodating

3

Collectivism

4

Hierarchy

Note: M = Mean SD = Standard deviation

Proper behavior Being tolerant Indirectness in dealing with others Indirectness in dealing with elders Respect to elders Being sensitive toward others Being flexible Appreciate of differences Majority consensus Teamwork Duty Advice Community concern Family relationship Wealth Obedient Power distance

M

SD

5.78 5.98 5.48 5.76 6.17 5.66 5.80 5.92 5.30 5.38 5.72 5.43 5.47 5.36 4.78 5.03 5.00

1.234 1.157 1.343 1.281 1.103 1.356 1.220 1.088 1.290 1.429 1.202 1.334 1.102 1.289 1.716 1.446 1.532

214

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Age 51-60 Income > 5000

Income 3001-4000 Male Income 4001-5000 Age 40-50 P2-L MelakaAge 29-39

Kedah

Income 2001-3000 Income 1000-2000 C2-H C6-H H2-H P5-LP1-L A3-L P3-1 H5-1 H6-H P4-L Pahang C1-1 C3-0 A1-1 H1-1 Johor C4-L C4-1 A2-L A2-1 P4-1 C3-1 H1-L A1-L P1-1 A3-H Selangor C1-L P5-H P2-H P3-L C6-L Trengganu Kelantan H2-L H5-L C2-LH6-L Sarawak Female Perak Sabah

Petaling

Penang Age 18-28

Income < 1000

Figure 14.3  Correspondence analysis output for Malays’ cultural values scales and demographic categories Note: Cultural values are marked in red; gender in blue; age in green; place in black; income in purple. Split into H and L indicates high- versus low-value orientation.

and geographic factors in reflecting Malays’ values. Malays with a higher income level clearly indicated a higher value orientation. Finally, the level of value orientation also increased with age. Overall, value orientation was somewhat higher for females and for those who were older and richer, which indicates that the instrument is sensitive enough to reflect value differences within different demographic categories in Malay culture.

THE USEFULNESS OF MIXED METHODS RESEARCH IN MEASURING INDIGENOUS CULTURAL VALUES The purpose of this chapter was to illustrate how mixed methods research can be applied as a rigorous method for developing a quantitative instrument responsive to an indigenous culture. A three-round Delphi study and field test was carried out by the research team to ensure the validity

and applied utility of the instrument. Representative samples from all over Malaysia were used for the standardization of the study and to identify Malay values. This comprehensive approach provides the basis for examining the structure of Malay values. Four values dimensions were extracted from the study—Politeness, Collectivism, Accommodation and Hierarchy. The indigenously derived instrument proposed value dimensions such as politeness and being accommodating that did not load on any universal quantitative instruments. At the most abstract level, politeness may indeed be a universal phenomenon (Brown & Levinson, 1987). However, it would be a grave error to assume that the values attached to politeness are similar across cultures. Indeed, what counts as polite behavior is different from one culture to another culture (Brown & Levinson, 1978). Malays’ politeness ideological notions are influenced by the concept of budi, which can be translated as “wisdom” or “intellect” (Lim, 2001). In Malay culture, the achievement of the politeness value types in an individual’s social world is one of the most

INDIGENOUS CULTURAL VALUES INSTRUMENT DEVELOPMENT

significant goals of everyday living, such that the concern for achieving politeness appears to be an even higher value than other important outcomes. The types of the politeness values identified in this study indicate the dynamic process to achieve harmony within a society that is both collectivist and hierarchical. This is more than simply just a term for “proper behaviour”, which is often mentioned in universal scales, such as Schwartz’s (2003) Value Scale, to indicate the politeness level across cultures. Accommodating values revealed one of the most unique characteristics of the Malay culture. It refers to the series of actions by and beliefs of the individual and community to accept and to be accepted by other people and otherness. The emphasis on being accommodating is manifested by one’s value of appreciating differences and being flexible in dealing with other people. It helps Malays to have a higher level of life satisfaction, a higher level of perceived health, and a lower level of depression. The value dimensions such as collectivism and hierarchy somehow are like universal scales. The mixed methods also allowed for the development of a theory-based understanding of values instead of just as an instrument. It also determined top values and returned to prior phases to revise the instrument. Finally, as indicated in the study, this method enhanced the convergent validity, reliability, and generalizability of the results. Therefore, future research should consider the profound impact of mixed methods research on the development of an indigenous cultural instrument.

CONTRIBUTION FOR USING MIXED METHODS RESEARCH IN DEVELOPING CULTURAL VALUES INSTRUMENT The mixed methods approach allows the development of instruments based on the inclusion of indigenous participants and consultation with local experts (for a discussion of indigenous partnerships in mixed methods research, see Chapter 15, this volume). In this research, there was a clear rationale for the use of survey data, list data, and openended data in addressing the same substantive question of what are our target culture’s values. Three further procedures are units of analysis, sampling, and instrumentation and data collection. Here, the units of analysis were integrated in the sense that the qualitative open-ended data and quantitative survey data had the same points of reference—these were experts and people from the same culture. The data-collection instruments of all

215

methods were linked to emphasize the complementary use of each method and to corroborate the findings from prior stages by addressing the same substantive issue of our target culture values. An example of this is to be found where the findings from the open-ended and list data were integrated to provide a fuller picture, and to produce a comprehensive list of values responsive to the target culture. The list of values was corroborated through quantitative survey data. The final procedure for achieving integration is the analytic strategies. Here, the analysis began by observing distinct techniques that were conventionally associated with each method; these were a descriptive and consensus analysis for the list and open-ended data. The qualitative inquiry was used to develop the theory-based categorization of values by generating formative items. It is hoped that the efforts described here might be replicated in the development of formative indicators in varied settings. Thereafter, a correspondence analysis of scale was used for the quantitative survey data. It allowed a more nuanced account and explanation of the target culture’s values at different layers of demographic groups, and the tracing of cumulative cultural development layer after layer. Finally, the results show the value of Delphi groups as part of a mixed methods research design in advancing validity and reliability of a quantitative cultural values instrument. The Delphi literature has several distinct gaps in methodological guidance. Some of the gaps in the literature for using the qualitative Delphi include how researchers should approach data reduction and analysis in qualitative Delphi studies (Brady, 2015). As demonstrated, this study offers a solution by describing how qualitative inquiry, which is suited for investigation of cultural values, can be used to inform instrument development, data reduction, and analysis in Delphi studies. It is hoped that the efforts described here might be replicated in varied settings.

WHAT TO READ NEXT Lenartowicz, T., & Roth, K. (1999). A framework for culture assessment. Journal of International Business Studies, 30(4), 781–798.

This article proposes a framework by which valid cultural groupings may be assessed. They indicate culture as a complex matter which cannot simply be understood at the national level. Onwuegbuzie, A. J., Rebecca, M. B., & Judith, A. N. (2010). Mixed research as a tool for developing

216

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

quantitative instruments. Journal of Mixed Methods Research, 4(1), 56–78. https://journals.sagepub. com/doi/10.1177/1558689809355805

This used mixed methods research as a tool for developing quantitative instruments and presenting a meta-framework—comprising several frameworks and models, as well as multiple research approaches—that they call an Instrument Development and Construct Validation (IDCV) process for optimizing the development of quantitative instruments. Taghipoorreyneh, M., & de Run, E. C. (2020). Using mixed methods research as a tool for developing an indigenous cultural values instrument in Malaysia. Journal of Mixed Methods Research, 14(3), 403– 424. https://doi.org/10.1177/1558689819857530

This study shows how mixed methods can be applied as a rigorous method for developing instruments to measure individual differences in personal cultural orientations. It illuminates the value of Delphi groups as part of a mixed methods research design and the utility of mixed methods research in advancing the validity and reliability of an indigenous cultural values instrument. Chapter 15 of this volume by Peter Rawlins, Philippa Butler, Spencer Lilley and Maggie Hartnett examines the principles of Aotearoa New Zealand’s indigenous research method: Kaupapa Maori ¯ research. M¯aori are tangata whenua (indigenous people) of our land and partners in Te Tiriti o Waitangi (The Treaty of Waitangi). The chapter introduces how Western and indigenous research methods might be “braided” together to impact research designs.

REFERENCES Adair, J. (2006). Creating indigenous psychologies: Insights from empirical social studies of the science of psychology. In U. Kim, K.-S. Yang, & K.-K. Hwang (Eds.), Indigenous and cultural psychology: Understanding people in context (pp. 467–485). Springer. Addison, T. (2003). E-commerce project development risks: evidence from a Delphi survey. International Journal of Information Management, 23(1), 25–40. Asma, A. (1992). The influence of ethnic values on managerial practices in Malaysia. Malaysian Management Review, 27, 318. Asma, A. (2001). The influence of Malay cultural values on management in Malaysia. Unpublished PhD thesis. Universiti Kebangsaan Malaysia, Malaysia.

Bearden, W. O., Netemeyer, R. G., & Teel, J. E. (1989). Measurement of consumer susceptibility to interpersonal influences. Journal of Consumer Research, 15, 473–481. Brady, S. R. (2015). Utilizing and adapting the Delphi method for use in qualitative research. International Journal of Qualitative Methods, 14(5), 1–6. Briedenhann, J., & Wickens, E. (2007). Developing cultural tourism in South Africa: Potential & pitfalls. In Richards, G. (Ed.), Cultural Tourism: Global and Local Perspective (pp. 69–90). The Haworth Hospitality Press. Brown, P., & Levinson, S. C. (1978). Universals in language usage: Politeness phenomena. In Questions and politeness: Strategies in social interaction (pp. 56–311). Cambridge University Press. Cheung, F. M., Van de Vijver, F. J. R., & Leong, F. T. L. (2011). Toward a new approach to the study of personality in culture. American Psychologist, 66(7), 593–603. Church, A. T. (2001). Personality measurement in cross-cultural perspective. Journal of Personality, 69(6), 979–1006. Delbecq, A., Van de Ven, A. and Gustafson, D. (1975). Group techniques for program planning: A guide to nominal group and Delphi processes. Scott-Foresman. Fontaine, R., & Richardson, S. (2005). Cultural values in Malaysia: Chinese, Malays and Indians compared. Cross-Cultural Management, 12, 63–77. Garson, G. D. (2014). The Delphi method in quantitative research (Statistical Associates Blue Book series) [Kindle version]. Retrieved from Amazon.com Geertz, C. (1973). The interpretation of cultures. Basic Books. Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008–1015. Haviland, W. A. (1990). Cultural anthropology (6th ed.). Holt, Rinehart & Winston. Hayes, T. (2007). Delphi study of the future of marketing of higher education. Journal of Business Research, 60, 927–931. Ho, D.Y.F. (1996). Filial piety and its psychological consequences. In M. H. Bond (Ed.), Handbook of Chinese psychology, (pp. 155–165). Hong Kong: Oxford University Press. Hofstede, G. (1980). Cultures and organizations: Software of the mind. McGraw-Hill. Hoy, R. R. (1993). A ‘model minority’ speaks out on cultural shyness. Science, 262(5136), 1117–1118. Hui, C. H., & Triandis, H.C. (1989). Effects of culture and response format on extreme response style. Journal of Cross-Cultural Psychology, 20(3), 296–309. Inglehart, R. (2003). Human values and social change: Findings from the values surveys. Brill.

INDIGENOUS CULTURAL VALUES INSTRUMENT DEVELOPMENT

Kim, U., Yang, K. S., & Hwang, K.K. (2006). Indigenous and cultural psychology: Understanding people in context. Springer. Lee, K. H., & Tan, C. B. (2000). The Chinese in Malaysia. Oxford University Press. Lenartowicz, T., & Roth, K. (1999). A framework for culture assessment. Journal of International Business Studies, 30(4), 781–798. Lim, L. (1998). Cultural attributes of Malays and Malaysian Chinese: Implications for research and practice. Malaysian Management Review, 33, 81–88. Lim, L. (2001). Work related values of Malays and Chinese Malaysians. International Journal of Cross-Cultural Management, 1, 209–226. Linstone, H. A. and Turroff, M. (2002). General applications: Introduction. In H. A. Linstone, and M. Turroff (Eds.), The Delphi approach: Technique and applications. Addison-Wesley. Ludwig, B. (1997). Predicting the future: Have you considered using the Delphi methodology? Journal of Extension,35, 1–4. Martino, J. P. (1972). Technological forecasting for decision making. American Elsevier, New York. McCrae, R. R., Terracciano, A., & 78 members of the Personality Profiles of Cultures Project. (2005). Universal features of personality traits from the observer’s perspective: Data from 50 cultures. Journal of Personality and Social Psychology, 88(3), 547–561. Miller, G. (2001). The development of indicators for sustainable tourism: results of a Delphi survey of tourism researchers. Tourism Management, 22(4), 351–362. Muthukrishna, M., Bell, A. V., Henrich, J., Curtin, C. M., Gedranovich, A., McInerney, J., & Thue, B.

217

(2020). Beyond Western, Educated, Industrial, Rich, and Democratic (WEIRD) psychology: Measuring and mapping scales of cultural and psychological distance. Psychological Science, 31(6), 678–701. Purcell, V. (1965). Malaysia. Thames & Hudson. Putnam, J.W., Spiegel, A.N., & Bruininks, R.H. (1995). Future directions in education and inclusion of students with disabilities: a Delphi investigation. Exceptional Children, 61, 553–576. Rokeach, M. (1973). Beliefs, attitudes and values: A theory of organization and change. Jossey-Bass. Schwartz, S. H. (1999). Beyond individualism/collectivism: New cultural dimensions of values. In U. Kim, H. C. Triandis, C. Kagitcibasi, S.-C. Choi, & G. Yoon (Eds.), Individualism and collectivism: Theory, method, and applications (pp. 85–119). Sage. Schwartz, S. H. (2003). A proposal for measuring value orientations across nations. Retrieved from: www.europeansocialsurvey.org/docs/methodology/core_ess_questionnaire/ ESS_core_questionnaire_human_values.pdf Sobaih, A.E., Ritchie, C., & Jones, E. (2012). Consulting the oracle? Applications of modified Delphi technique to qualitative research in the hospitality industry. International Journal of Contemporary Hospitality Management, 24, 886–906. Taylor, R. E., & Judd, L. L. (1989). Delphi method applied to tourism. In S. Witt, & L. Moutinho, (Eds.). Tourism marketing and management handbook. Prentice Hall. Triandis, H. C. (1996). The psychological measurement of cultural syndromes. American Psychologist, 51, 407–415.

15 What Can Mixed Methods Partnerships Learn from Kaupapa M¯aori Research Principles? P e t e r R a w l i n s , P h i l i p p a B u t l e r, S p e n c e r L i l l e y and Maggie Hartnett

INTRODUCTION Culturally relevant approaches to knowledge creation require that indigenous ways of knowing and doing are valued (A. Macfarlane & Macfarlane, 2019). Although Aotearoa New Zealand is a legally bicultural country, too often indigenous ways of knowing are subsumed by more dominant western values, resulting in a knowledge base where M¯aori (New Zealand’s indigenous population) voices are often unheard. This exacerbates the lack of equality that is all too common in culturally diverse communities (Chilisa, 2020; Chilisa & Tsheko, 2014; A. Macfarlane, 2012). Research is influenced, consciously or unconsciously, by a set of philosophical assumptions about how the world works, how knowledge is created, appreciated and shared (Mertens, 2015). Within the field of mixed methods research, there have been calls to identify non-western worldviews and indigenous philosophies that can impact how mixed methods research is conceptualised and conducted (Fetters & Molina-Azorin, 2018). Examples of such frameworks include the Asian philosophy of Yinyang, where the duality of subjective and objective perspectives are seen as a way of unfolding and coordinating multidimensional relationships that are complex and changing

(Wang 2012). Similarly, the Pacific research frameworks of fa’afaletui (Goodyear-Smith & ‘Ofanoa, 2022) and talanoa (Gremillion et al., 2021) draw on the concepts of connectiveness and a holistic approach to weave together different perspectives to create new knowledge. In Aotearoa New Zealand, an indigenous research paradigm called “kaupapa M¯aori research” has been developing over the past 30 years. The question remains, how can these indigenous research paradigms impact on, and add to, how mixed methods research is conceptualised and conducted? In Aotearoa New Zealand, a metaphor that has been used to describe the coming together of two distinctive research approaches is He Awa Whiria, or the braided river (S. Macfarlane et al., 2015; Martel et al., 2022). In this metaphor, two worldviews, one a western research approach and the other a M¯aori approach, are represented by two streams of a braided river that flow independently of each other, but converge from time to time to create a more powerful flow of water, before diverting off again, only to converge at a later point in the river. Each of the streams is a legitimate river in its own right and both journey towards a common goal, but, rather than being isolated, independent and competing with each other, they become relational or braided. Combining the strengths of the two worldviews creates better

WHAT CAN MIXED METHODS PARTNERSHIPS LEARN FROM KAUPAPA MĀORI RESEARCH? 219

understanding and better outcomes for indigenous and non-indigenous individuals and groups (A. Macfarlane & Macfarlane, 2019). This chapter argues that, in order for this braiding to occur, a greater understanding of the cultural principles inherent in indigenous research methods must be reached. Although the He Awa Whiria metaphor has generally been applied to the “braiding” of Western and indigenous research paradigms, it could equally apply to mixed methods research more broadly. Rather than indigenous and non-indigenous ways of knowing being the two braids, it is qualitative and quantitative research approaches that are the two braids. Although both are legitimate ways of conducting research, the “braiding” of the two approaches forms a more powerful approach than either alone. This has been described in the mixed methods literature as the 1 + 1 = 3 phenomenon where integration of qualitative and quantitative methods produces a whole that is more than the sum of the parts (Fetters & Freshwater, 2015). This chapter seeks to illustrate how a greater understanding of indigenous worldviews can positively influence mixed methods research design through a He Awa Whiria approach to research. To do this, we will unpack the principles underpinning one indigenous research approach in Aotearoa New Zealand: kaupapa M¯aori research. Although using an indigenous research framework to make our argument, it is important to note that we are not arguing that the braided river approach is an indigenous research paradigm per se. The goal of indigenous research paradigms is decolonisation and indigenisation. They seek to “bring to the center of the research process marginalised voices subjected to exploitation and abuse through experiments and colonising research” (Chilisa & Tsheko, 2014, p. 222). In a sense, they seek to

Figure 15.1  A braided river Source:(CCO Public Domain)

transform the lives of marginalised people. Indeed, links have been made between indigenous research paradigms and transformative research paradigms (Mertens & Cram, 2016). What we are arguing in this chapter is that a mixed methods research approach that understands the principles underpinning indigenous research paradigms—in our case, a kaupapa M¯aori research approach—can, through a He Awa Whiria approach to research, produce stronger research designs. But first, we need to understand the principles underpinning kaupapa M¯aori research. This chapter is organised into five parts. We first present a brief history of the complicated relationship between M¯aori and non-M¯aori dating back more than 180 years. This brief history is important, as it helps explain why M¯aori have typically had an inherent distrust of western ways of being and doing, leading to M¯aori being underrepresented in western research approaches (A. Macfarlane & Macfarlane, 2019; L. T. Smith, 2012). Second, we explain three approaches to research involving M¯aori in order to distinguish between kaupapa M¯aori approaches and other research approaches involving M¯aori. Third, we deconstruct a number of major M¯aori cultural principles associated with kaupapa M¯aori research and consider how understanding them might inform crucial parts of mixed methods research designs. Fourth, parallels between kaupapa M¯aori research and community-based research are considered, to demonstrate how the underlying principles of kaupapa M¯aori research can apply more broadly. We then offer two examples of research projects to help illustrate how the consideration of important indigenous principles in research design will help to capture the voices of indigenous and non-indigenous peoples. The first of these is a research project conducted by Bishop et al. (2009). This project

220

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

was conducted using a kaupapa M¯aori approach and we include it to illustrate the key cultural principles embedded in this approach. The second research project was conducted in one Aotearoa New Zealand university during the Covid pandemic (Hartnett et al., 2020). This project can be classified as “research involving M¯aori” and illustrates how kaupapa M¯aori principles can inform the design of any research project so that it can more fully draw on underrepresented indigenous voices that are often not present in mixed methods research. The chapter ends with a discussion of our main points and a conclusion.

FOUNDING OF AOTEAROA NEW ZEALAND: SITUATING A COMPLICATED PAST The colonisation of Aotearoa New Zealand has created significant tension between M¯aori and non-M¯aori, resulting in resentment towards the colonisers and disengagement by M¯aori in western research approaches (A. Macfarlane & Macfarlane, 2019; L. T. Smith, 2012). Understanding the history of interactions between M¯aori and colonisers helps to recognise and redress these tensions. Unfortunately, Aotearoa New Zealand’s story is not unique, with similar stories of colonisation found in many indigenous cultures (Chilisa, 2020). Aotearoa New Zealand is a young country in terms of both geography and population. Historians believe that the M¯aori people first arrived in New Zealand about 1,000 years ago from Eastern Polynesia. The Dutchman Abel Tasman was the first European to visit New Zealand in 1642, but it wasn’t until after Englishman James Cook visited the country in 1769 that Europeans began to arrive in greater numbers. Whalers and sealers were the first Europeans to arrive, followed by traders and those looking to settle on the land. By the 1830s, the Bay of Islands district, in the north of New Zealand, had established itself as a trading hub and was well known for its ability to provide fresh water and food to whale-boat captains and their crews. Unfortunately, with drink and women readily available, it became known for drunkenness and prostitution, which frequently culminated in brawling and other illegal behaviours (King, 2003). As the European population of New Zealand grew, so did the need for land and other resources, resulting in growing tensions between M¯aori and non-M¯aori. As a result, 13 chiefs of the Ng¯apuhi (a M¯aori tribe in northern New Zealand) wrote to King William IV of the United Kingdom in 1831,

seeking an alliance and protection from other foreign powers. Following this letter, James Busby was appointed as the British resident in the Bay of Islands, with the responsibility of controlling the behaviour of the Europeans. In addition to these responsibilities, Busby assisted rangatira (chiefs) in asserting their independence from outside meddling. Busby was driven by his fears about increasing French involvement in northern New Zealand, particularly Baron Charles de Thierry, who had purchased significant swaths of land (Walker, 2004). This was in preparation for him to establish his own French colony, of which he proposed being the sole ruler. The British Crown recognized the sovereignty of the Confederation of United Tribes (Te Whakaminenga) in the 1835 Declaration of Independence—, He Whakaputanga—which was signed by 52 chiefs (Orange, 2015). M¯aori saw this declaration as a method to deal with the obstacles provided by European contact, to cement their alliance with the United Kingdom, and to demonstrate their power in the wider world. The Declaration further stated that Te Whakaminenga had sovereign power and jurisdiction over the land, and that no strangers could enact laws. Despite the fact that He Whakaputanga did not represent the majority of M¯aori, it is often regarded as a foundation for the assertion of indigenous rights in Aotearoa New Zealand.

Te Tiriti o Waitangi: Aotearoa New Zealand’s Founding Document Despite He Whakaputanga, there was still a lot of demand for land and other resources, as New Zealand became a more appealing alternative for those from the United Kingdom who wanted to acquire land. By 1839, it had become clear that the Crown needed to take measures to ensure that European anarchy could be controlled and M¯aori rights could be preserved. As a result, the Treaty of Waitangi (Te Tiriti o Waitangi) was negotiated and signed in 1840 by representatives of the British Crown and M¯aori chiefs. It is worth noting that the actual text was written in English and then swiftly translated into te reo M¯aori (M¯aori language). It is suspected that specific words and phrases were modified throughout the translation process to make them more appetising to M¯aori, making them more likely to sign (Walker, 2004). As a result of these alterations, the English and M¯aori versions of Te Tiriti have very different meanings. These adjustments were considered sensible at the time, based on recommendations from colonial authorities who were familiar with

WHAT CAN MIXED METHODS PARTNERSHIPS LEARN FROM KAUPAPA MĀORI RESEARCH? 221

M¯aori customs. Today, these adjustments appear to be inept at best, and deceptive at worst. Despite the fact that more than 500 chiefs signed Te Tiriti, the English version received only 39 of these signatures. The majority of M¯aori signed the te reo M¯aori version. Disagreements regarding the two versions’ intent and interpretation are at the root of ongoing conflicts between the two groups, as well as M¯aori mistrust of the Crown and those who represent it (Walker, 2004). To give an example of these tensions, there was a major variance in understanding of the word ‘kawanatanga’. This was used as a synonym for “sovereignty” by Europeans and interpreted by M¯aori as “governance.” The line “tuku rawa atu ki te Kuini o Ingarangi ake tonu atu ki te kawanatanga katoa o o ratou wenua” appears in the M¯aori version of Te Tiriti. This exactly translates to the chiefs as: “give absolutely to the Queen of England for ever the complete government over their land” (Kawharu, 1989). In English, the text reads “Cede to Her Majesty the Queen of England totally and without reservation all the rights and powers of sovereignty”. The difference between these two phrases is significant, as sovereignty and government are two very different concepts. The use of kawanatanga to represent sovereignty in the te reo M¯aori version (when it would have been understood as governance) meant that the intentions of the Crown were misrepresented to those chiefs, as they were unlikely to sign sovereignty over to the Queen. This distinction may appear insignificant from a western point of view. However, it is very significant when examined through the perspective of a M¯aori worldview, te ao M¯aori. The distinction lies in how the concept of whenua (land) is understood. Whenua refers to both land and placenta in M¯aori. The placenta and pito (umbilical cord) are buried on wh¯anau (family) property after birth, establishing a spiritual relationship between the infant and the land (Mead, 2016). Due to their common descent from Ranginui (Sky father) and Papat u¯ a¯ nuku (Earth mother), whakapapa (ancestral links) is at the heart of the relationship between whenua and all other living creatures in te ao M¯aori. As a result, tangata (people) and whenua (land) belong together, and their relationship is alive and well— hence, the term tangata whenua (people of the land). M¯aori operate as guardians (and not owners) of the land to ensure that it is nurtured and preserved for future generations, and although they may grant others access to and use of the land and its resources, the land and its resources will revert to them when they are no longer needed. For the British, land was a commodity that could be measured and divided into sections that could be bought and sold over and over again. Between the

two groups, the concept of ownership was significantly different. “Ko te atarau o te whenua i riro a te Kuini, ko te tinana o te whenua i waiho ki ng¯a M¯aori,” said N¯opera Pana-kareao in 1840: “The Queen receives only the shadow of the land; the substance remains with us” (Orange, 2020, p. 49). Since its inception, Te Tiriti has defined the relationship between M¯aori and non-M¯aori. As this brief history demonstrates, this has been characterized at various times by conflict, injustices, and mistrust. Despite the pledges of Te Tiriti o Waitangi, “the colonialism of Aotearoa/New Zealand and subsequent neo-colonial dominance of majority interests in research has remained” (Bishop, 1999, p. 1). As a result, many M¯aori continue to fear what they perceive to be western methods of thinking. This extends to a distrust of research that is perceived to be based on western ideologies (L. T. Smith, 2012).

RESEARCH WITH MĀORI During the nineteenth and early twentieth centuries, Europeans in New Zealand had a fascination with studying M¯aori culture and customs (L. T. Smith, 2012). These studies treated M¯aori as the object of research and the findings were analysed from a western perspective. As a result, data were misinterpreted and unique cultural nuances were missed. The complexity of M¯aori knowledge was misrepresented by such research, which instead attempted to establish the “superiority” of European knowledge, language, and culture. M¯aori strongly critique this approach to research and view it as just another way of being colonized (Bishop, 1999; L. T. Smith, 2012). In response to these criticisms of research on M¯aori (Bishop, 1999; L. T. Smith, 2012), a M¯aorifocused research tradition was developed in the 1990s that promotes te ao M¯aori (M¯aori worldview) at its centre. The validity of earlier research was questioned due to the tendency for it to have been conducted and analyzed using the nonM¯aori researchers’ own cultural worldview, or Europeans solving “indigenous problems” using a lens that was created using deficit theory as its basis (Bishop, 1999). Instead, the focus shifted to research with or by M¯aori. At the heart of this new M¯aori-focused research tradition was the development of M¯aori research methodologies, created and applied in three distinct frameworks: “research involving M¯aori”, “M¯aori-centred research” and kaupapa M¯aori research (Bishop, 1999; Cunningham, 2000; G. H. Smith, 2015). The three approaches are

222

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

distinguished by the amount of M¯aori involvement in their application and by the knowledge that is generated from the research. Table 15.1 compares these three approaches, which are discussed in detail below. Of the three methods, “research involving M¯aori” has the smallest level of M¯aori involvement (Cunningham, 2000; P u¯ taiora Writing Group, 2010). This might mean that there are M¯aori participants and/or researchers, but they are not the primary focus of the research being conducted, M¯aori are not the instigators of the research, nor are the results likely to be analysed through a M¯aori cultural lens. The knowledge that is produced will be mainstream knowledge about M¯aori (Cunningham, 2000). The benefit to M¯aori from these types of projects is likely to be quite low. “M¯aori-centred research” requires a higher involvement from M¯aori researchers (P u¯ taiora Writing Group, 2010). This would include M¯aori as leaders or in another senior role within the research team. It would be expected that a significant number of the participants are M¯aori. The results of the research would use M¯aori perspectives in the analysis, but would include non-M¯aori concepts as well. One of the core advantages of using a M¯aori-centred approach is that it provides a research team with the ability to choose a cross-cultural set of methodologies and analytical lenses suitable for the collection and interpretation of data in that context (Cunningham, 2000). The combination of M¯aori and non-M¯aori researchers enables the team to draw on the most appropriate methods for conducting the research, with M¯aori members of the team ensuring that they will lead to results and outcomes beneficial for the M¯aori participants (as well as the non-M¯aori participants). M¯aori-centred research will lead to the production of M¯aori knowledge, “albeit measured against largely mainstream standards and

methodologies (for example, refereed journal articles)” (Cunningham, 2000, p. 65). Kaupapa M¯aori research can be described as M¯aori research conducted with M¯aori, by M¯aori researchers, using a M¯aori lens (L. T. Smith, 2015). Kaupapa M¯aori research is developed in a te ao M¯aori worldview. Within te ao M¯aori, the concept of kaupapa M¯aori embraces the validity and legitimation of M¯aori language, knowledge, and culture (G. H. Smith, 2015). This includes the focus of the research being of high value to M¯aori rather than the researchers, or research that is identified and co-designed by a M¯aori community or organisation to assist them to address an issue or problem. The project team will be led by a senior researcher who is M¯aori, and the remainder of the team members will be predominantly M¯aori. Occasionally, the team will include members from the community who will be mentored by senior researchers, with a view to building the research capacity of that community, so they can conduct their own research in the future. Critical to kaupapa M¯aori methods is the use of values that encompass tikanga M¯aori (M¯aori customs) and the analysis of research results using a lens that is distinctly M¯aori in its focus. The outcomes of such research become m¯atauranga M¯aori (M¯aori knowledge) and should first benefit the community participating in the research, and then wider M¯aori society (Cunningham, 2000). Regardless of the choice of approach taken to conduct M¯aori research, critical to all three is the need for the development of a strong relationship between the research team and the communities they are working with. These relationships should be ongoing. Essential to this is the concept of kanohi kitea (the face seen), where research team members are frequent and active participants in community events and affairs. Developing these relationships ensures that trust builds between community members and researchers. Such

Table 15.1  Three approaches to research with Māori Research involving M¯aori

M¯aori-centred research

Kaupapa M¯aori research

Focus Initiation Researchers Participants Analysis

Non-M¯aori interests Non-M¯aori Predominantly non-M¯aori Some M¯aori involvement Non-M¯aori lens

M¯aori interests M¯aori Predominantly M¯aori M¯aori Te ao M¯aori lens

Ownership of results Knowledge generated Benefits

Non-M¯aori Mainstream knowledge Non-M¯aori communities

Cross-cultural M¯aori or non-M¯aori High involvement of M¯aori Significant number of M¯aori M¯aori and non-M¯aori perspectives and concepts M¯aori or non-M¯aori M¯aori knowledge M¯aori and non-M¯aori communities

M¯aori M¯aori knowledge M¯aori communities

WHAT CAN MIXED METHODS PARTNERSHIPS LEARN FROM KAUPAPA MĀORI RESEARCH? 223

connections will lead to a relationship embodying the principle of whakapapa (kinship built on shared interests). These relationships will be further strengthened through other tikanga M¯aori practices, particularly those of manaakitanga (respect), whanaungatanga (connecting), whakamana (giving prestige to), and tauutuutu (reciprocity).

PRINCIPLES OF KAUPAPA MĀORI RESEARCH With its emphasis on delivering M¯aori-focused outcomes, kaupapa M¯aori research aims to make a positive difference for M¯aori. Linda Tuhiwai Smith (2012) cautions that, in New Zealand’s research landscape, it is a methodological approach that must be led by M¯aori researchers. Although non-M¯aori can be involved, they cannot conduct kaupapa M¯aori research on their own, but can participate in a research team by partnering with and being led by M¯aori researchers. Critical to conducting a successful kaupapa M¯aori research project is the use and application of underlying principles. These principles are fluid and changeable (Mahuika, 2015), but there are commonly agreed upon core principles. We focus here on the principles of mana, rangatiratanga, whakapapa, whanaungatanga, te reo, and tikanga M¯aori (L. T. Smith, 2015, 2021; www.rangahau.co.nz/rangahau/). These values are described in more detail below, along with an explanation of how they can be applied to research. Mana can be defined as the prestige, dignity or status associated with an individual (Barlow, 1991). Mana can be inherited from ancestors or earned by individuals and bestowed through recognition by others. In a research context, mana recognises that participants have a right to be informed, be confident that their participation will contribute to research outcomes, and have the power to decide how they engage with the research process. The research process also provides an opportunity for an individual to gain new skills and knowledge, which can be mana enhancing. Rangatiratanga involves the exercise of sovereignty, chieftainship, leadership, self-determination, and self-management (Mead, 2016). In applying this principle to research, it provides a community with the right to determine the direction a research project takes and how they will engage with it. As a community, there is an expectation that they will be partners with the research team. Whakapapa is normally related to identity and descent that is associated with genealogy

and connects people to the cosmology, gods, and to their ancestors (Barlow, 1991). However, in a research context it can be applied much more broadly. It relates to the idea that research should recognise and value the importance of the community and its distinct identity. Whanaungatanga is about establishing connections, developing relationships, and kinship (Mead, 2016). It involves creating shared experiences and collaborating to generate a sense of belonging. Through connecting “kanohi ki te kanohi” (faceto-face), researchers and community can ensure that their shared relationships result in effective research and successful outcomes. The principle of te reo, or “language”, can be thought of in practical terms — that research should be conducted using the language of the community. This, L. T. Smith (2015) believes, should include development of better quality bilingual resources, consent forms, and information sheets, and the employment of researchers who are skilled in this area. This could include a requirement for all research documents, such as questionnaires, to be available in te reo M¯aori (and the correct dialect where appropriate). More broadly, the research experience should be supported by shared understandings of the purposes and outcomes of the research. The link between language and knowledge are critical, as knowledge is often only truly understood in the language in which it was created. Tikanga M¯aori is defined by Mead (2016) as “the ethical and common law issues that underpin the behaviour of members of wh¯anau, hap u¯ and iwi as they go about their lives and especially when they engage in the cultural, social, ritual and economic ceremonies of their society” (p. 15). It is important that when conducting research in a M¯aori context that the requirements of tikanga M¯aori are adhered to. The use of these ensures that the various stages of the research process are conducted in ways that are culturally correct for the community and are reflective of their understanding of the world, thus ensuring that their needs are fulfilled. To help ensure that M¯aori knowledge and principles are upheld in research conducted through a kaupapa M¯aori research approach, Bishop (1996) introduced a model that was later adapted by S. Macfarlane and Macfarlane (2018). The IBRLA models consist of five key components that can be used to guide researchers through a culturally relevant research design process: Initiation, Benefits, Representation, Legitimation, and Accountability. Within each of these five areas is a series of accountability questions that the researcher can use to ensure that M¯aori knowledge and principles are included throughout the research project.

224

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

We have added the corresponding kaupapa M¯aori research principle to illustrate how the elements of this model are consistent with these principles (see Figure 15.2). We have introduced the IRBLA model as an organisational tool used in our two examples below, to help the reader understand how kaupapa M¯aori principles may be operationalised in research designs.

Ini�a�on

Benefits

Representa�on

Legi�ma�on

Accountability

COMMUNITY-FOCUSED RESEARCH APPROACHES Kaupapa M¯aori research has grown locally out of the complex set of conditions and history described above. It is very specific to the New Zealand context. There are, however, strong parallels to other critical approaches to research that have developed elsewhere.

Accountability ques�ons • Who conceptualised and ini�ated this research project? • How did Māori par�cipate in the conceptualisa�on and ini�a�on process? • How was the agreement to proceed with the research achieved?

Corresponding kaupapa Māori research principles • Ranga�ratanga • Whanaungatanga

Accountability ques�ons • How will the research (process and outcomes) accrue benefits for Māori? • How has the informa�on been shared with Māori about the intended benefits? • How will these benefits be determined and measured and by whom?

Corresponding kaupapa Māori research principles • Mana • Whanaungatanga

Accountability ques�ons • Whose ideas will be represented in the methodology, design and approach? • How will Māori thinking and knowledge be represented at all research phases? • How will this be monitored so that ongoing agreement/partnership is maintained?

Corresponding kaupapa Māori research principles • Ranga�ratanga • Mana

Accountability ques�ons • Who will legi�mate the analysis and interpreta�on of informa�on/research data? • How will Māori understandings be legi�mately represented? • How will this be structured so that research fidelity is achieved/protected?

Corresponding kaupapa Māori research principle • Whakapapa

Accountability ques�ons • Who is accountable to whom – and in what ways? • How will ongoing and mutual accountability be built into the research process? • How will this be monitored and evaluated to ensure safety for all stakeholders?

Corresponding kaupapa Māori research principle • Tikanga Māori

Figure 15.2  The IBRLA framework and corresponding kaupapa M¯aori research principles

WHAT CAN MIXED METHODS PARTNERSHIPS LEARN FROM KAUPAPA MĀORI RESEARCH? 225

These critical research approaches have emerged from an understanding that any research project involves issues of power between the researcher and the researched. They are a reaction to the concept of research where the power and privilege in the relationship belongs to the researcher, and the community is the object of research; where the researcher is the primary beneficiary of the research, and the community gets little benefit in return (Torre et  al., 2018). These approaches challenge the basic premises of academic research—that it is objective, detached, and conducted by experts for an audience of other experts (Root, 2007). Instead, the desire is to distribute power and decision-making more equally between researcher and participants. For a discussion about leveraging mixed methods communitybased participatory research approaches in diverse social and cultural contexts to advance health equity, see Chapter 29 (this volume). Collectively, these approaches are known as community-based research, or by other terms such as collaborative research, participative research, transformative research, or participatory action research. In this kind of research, the participants in the research are also co-researchers. Different approaches to community-based research may vary in the degree to which power is shared, but all subscribe to the basic understanding that research should be conducted with the community, for the benefit of the community (Root, 2007). In community-based research, the choice of topic and methods is shared between researcher and community co-researchers. The research is conducted in the language of the community, the data are interpreted in a way that is meaningful to the community, and the findings are published in media that the community has ready access to (Root, 2007). The participation of the community must be true participation at every stage of the research process—design, data collection, analysis, interpretation and dissemination. Merely involving community members as consultants runs the risk of being a tokenistic, box-ticking exercise (Torre et al., 2018). Community-based research draws on community expertise and community knowledge to address issues of concern for the community. Often, the community is one that is disadvantaged or marginalised in some way, and the issues are about social or political inequalities. The focus is on giving the community a voice on issues of importance to them, and enacting change for the community (Gomm, 2008; Torre et  al., 2018). Such research does not claim to be neutral (Gomm, 2008), but instead is actively transformative (Root, 2007). Mixed methods research that takes an ethical, inclusive stance to involve community

members and focus on issues that matter to those communities, can support such transformative change (Mertens, 2021). For further discussion of the use of a transformative lens in mixed methods design, see Chapter 4 (this volume). The parallels to kaupapa M¯aori research should be obvious: just as kaupapa M¯aori research arises out of issues important to M¯aori and seeks to benefit M¯aori communities, so too does communitybased research seek to benefit disadvantaged or marginalised groups. In an important way, however, kaupapa M¯aori research is a step beyond community-based research. A goal of communitybased research is to grow expertise from within the community. In Aotearoa New Zealand, through the mechanism of kaupapa M¯aori research, that expertise has been developing over the past few decades, and is located in universities, w¯ananga (M¯aori higher education organisations), iwi organisations, and government departments.

TWO ILLUSTRATIVE EXAMPLES The following examples provide snapshots of two research projects. The first study was carried out in New Zealand schools in the early 2000s and looked at ways to improve educational success for M¯aori in secondary schools (Bishop et al., 2009). We include it here as it is illustrative of a kaupapa M¯aori research project and shows how the IRBLA model can be operationalised. The second study (Hartnett et  al., 2020) was conducted within Massey University, our home institution, and is an example of research involving M¯aori. Although it was not specifically designed along kaupapa M¯aori principles, we include it here to illustrate how any research project might draw from these principles.

Example 1: Community-benefiting Educational Programme The first example is research carried out in New Zealand schools by Russell Bishop and his colleagues (Bishop et al., 2009). Critical to the success of the research project and the educational programme developed from its findings was the use of a kaupapa M¯aori research approach. The focus of this study was to determine how M¯aori students could be supported to have higher levels of educational achievement when they exit compulsory schooling. The research was initiated through observations by Bishop, an experienced

226

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

secondary teacher of M¯aori descent himself, and in consultation with other M¯aori educationalists, that M¯aori students were leaving school without fulfilling their educational potential. In 2001, the predominantly M¯aori research team started a major research project that aimed at finding pedagogical solutions to improve M¯aori student outcomes. The research team drew on Bishop’s (1996) IBRLA model to ensure that the participants could exercise greater control (rangatiratanga) over issues such as initiation (of the research), benefits (that accrue from the research), representation (of the participants’ voices), legitimation (of findings) and accountability (back to the research participants). In this way, they argued, principles underpinning kaupapa M¯aori research would be central to the research design. The principles underpinning the research were defined and identified by the research team as rangatiratanga (self-determination); taonga tuku iho (cultural aspirations); ako (reciprocal learning); kia pike ake o ng¯a raruraru o te kainga (strong school and home relationships); wh¯anau (establishment of strong relationships in the school); and kaupapa (development of strong vision). Although there are some differences to the principles we defined earlier, there are also similarities. This is an important part of understanding how kaupapa M¯aori principles are determined, as there is no one clear set of values and principles that apply to all situations. These values will often be determined by not only tikanga M¯aori, but also by the themes represented in the data that have been collected. In this particular study, there was also a major commitment to operationalising the guarantees made in Te Tiriti o Waitangi (Bishop et al., 2003), most noticeably to the three articles central to Te Tiriti: • Research would be conducted within partnership/power-sharing modes of decision making (Article One) (which relates to the kaupapa M¯aori principle of rangatiratanga). • M¯aori cultural aspirations, preferences and practices would guide the research (Article Two) (or the principle of whakamana). • Findings should contribute to the betterment of young M¯aori people in our schools (Article Three) (or the principle of whanaungatanga). The participants in the project were M¯aori students in Years 9 and 10 (approximately 13–15 years of age), and others involved in their education, including wh¯anau, principals and teachers, some of whom were not M¯aori. The study was a convergent parallel mixed methods study consisting of four threads. Three of the threads were qualitative, with the fourth being a quantitative thread.

Interviews with student participants were held first in focus group settings, and then with individuals who volunteered to be further involved. Participants included students whom schools had identified as either being engaged or not engaged with their schooling. Focus group interviews were held with wh¯anau, teachers and principals. Where appropriate, these took place at schools, as well as other venues, including marae (M¯aori community gathering places) and private homes. The researchers aimed to gain an understanding of the students’ educational experiences, from their perspective and those of their wider wh¯anau, and identify how these experiences could be improved to result in successful educational outcomes. In addition to the interviews and focus groups described above, the team also observed classroom practices of teachers and how these changed after intervention (through professional development activities). Using a whanaungatanga approach, a process of collaborative storytelling was established, integrating the three qualitative threads. This provided the research participants with a sense of rangatiratanga, as their voices formed the basis of the narratives, which gave their contributions legitimacy, and upheld the mana of the students participating in the research. The fourth thread involved access to the achievement and attendance data of the schools involved, and was used to determine whether noticeable improvements had occurred among the student participants. In developing the narratives, Bishop and his co-researchers placed the emphasis on reflecting the meanings that participants had given to their experiences, rather than interpreting it from the lens of the research team members. This was in keeping with their observation that past researchers had taken evidence gathered from participants and incorporated it with their own views and delivered research findings in a way that negated the voices of the participants and privileged the researcher’s own interpretation of that evidence. In this way, the ownership of the results remained with the participants and the communities they came from. Further use of these results to develop a professional development programme needed to be negotiated with the owners of this knowledge. The knowledge generated from this research was that M¯aori students and their wh¯anau believed that the quality of the interactions in class that students experienced with their teachers had a significant influence on their achievement. However, teachers and principals tended to suggest that the major influence on M¯aori educational achievement were the attitudes of M¯aori students and the circumstances of their families or wh¯anau. The benefit of the research was further enhanced by the creation of a professional development

WHAT CAN MIXED METHODS PARTNERSHIPS LEARN FROM KAUPAPA MĀORI RESEARCH? 227

programme aimed at teachers and schools (Te Kotahitanga), and through the development of an Effective Teaching Profile aimed at culturally responsive teaching practices. The rollout of Te Kotahitanga by the Ministry of Education, based on the findings of Bishop and his fellow researchers, was evidence of the value that kaupapa M¯aori research makes to providing M¯aori solutions for and by M¯aori. In addition to benefits to M¯aori, because of the wide rollout of this professional development opportunity and the changes to broader teacher practice, there were significant benefits to non-M¯aori as well. In terms of the IBRLA framework and the principles of kaupapa M¯aori research, the research was conceptualised and initiated by M¯aori (Initiation: rangatiratanga) and drew on relationships within schools (whanaungatanga). The professional development project that was subsequently rolled out across the country directly benefited M¯aori and non-M¯aori alike (Benefits: mana, whanaungatanga). The lead researcher and most of the research team were M¯aori, ensuring that the methodology, the design, and the approach to the research was based on M¯aori thinking and knowledge (Representation: rangatiratanga, mana). The whanaungatanga approach to collaborative story telling legitimated the analysis and interpretation of the data (Legitimation: whakapapa). The close relationship between the lead researcher, who is a significant figure in M¯aori educational research, and the wider M¯aori community ensured that accountability was enacted through a te ao M¯aori lens (Accountability: tikanga M¯aori).

Example 2: Institution-wide Research Project The second example focuses on research conducted in 2020 by three of the authors on the changes to assessment at Massey University brought about because of the COVID-19 pandemic (Hartnett et al., 2020). The research in this example can be described as research involving M¯aori. In March 2020, due to the COVID-19 global pandemic, the New Zealand Government introduced an alert level system and put the country into an initial six-week lockdown. The high degree of uncertainty around how long the country might be in lockdown and what might happen afterwards meant that schools and universities had to move to emergency remote teaching (ERT) to ensure the continuation of their teaching, learning, and assessment. As part of these changes, Massey University made the decision to replace all proctored paper-based examinations (which were

offered at various locations around the world) with a variety of technology-enhanced assessments. Prior to the pandemic, the university had been exploring greater use of online assessments and, with the forced changes brought about by COVID19, saw an opportunity to conduct research from within its own institution on student and staff perspectives of the change towards technologyenhanced assessments. The primary goal of this research was to influence internal decisionmaking around future policies and practices on assessment. The university commissioned a research team consisting of three of the authors. None of the researchers identify as M¯aori, yet they all value M¯aori perspectives. Additionally, as staff members, they had an existing relationship with the university and a vested interest in the findings of the study. All participants involved in the research were either students or staff at the university. A convergent parallel mixed methods research design was adopted involving three threads. The first comprised a survey of students who were impacted by the changes. The second involved a survey of academic staff who instigated changes to assessments. The third involved interviews with decision-makers throughout the university. The three data sets were analysed separately using standard, westernised thematic analysis and inferential statistics, and then integrated in the write-up of the final report. The two senior leaders within the university who had initiated the research (the Provost and the Deputy Vice Chancellor Students and Alumni) actively endorsed the project to the various participant groups. Students and academic staff members received a personalised email, inviting them to participate in the research. Our Deputy Vice Chancellor of Students and Alumni is M a¯ ori and was the designated person communicating with students during the COVID-19 pandemic. Students are used to getting messages from him and, as such, have an established relationship with him. A summary of the findings was made available to participants, and the results are being used to inform the university’s assessment policy and practices. One of the factors that surprised us in the research was the high response rate to the survey from M¯aori students. Traditionally, M¯aori and Pacific Peoples do not complete surveys, as they often have no prior relationship with the researchers and prefer to engage with people kanohi ki te kanohi (face to face) (L. T. Smith, 2012; A. Macfarlane & Macfarlane, 2019). Figure 15.3 highlights that the proportion of students who identified as M¯aori in the survey very closely approximated the proportion who identified as M¯aori in the wider university. This was also the

228

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 15.3  Ethnic groups of respondents in COVID assessment research compared with university figures Notes: A chi-square test revealed that there were more students who identified with an Asian ethnic group than expected in the survey sample (χ2 = 324.987, df = 4, pphen), where interview data would be collected primarily to elucidate statistical results and further describe pre-existing hypotheses. When the initial phenomenological findings proved to be data rich and highly illustrative during early analysis, the authors decided to make them more of a central focus of the study (quan + phen) (Figure 18.1). A qualitative and mixed

methodologist was recruited to enhance the rigour of the phenomenological strand and facilitate the integration of data. The team shifted from sequentially analyzing data toward a more simultaneous and integrated approach. Unanticipated qualitative findings informed patient-centred clinical recommendations, generated hypotheses, and produced novel directions for future work. This required more frequent team meetings, additional learning for clinical researchers new to phenomenology, and a more collaborative writing process than originally planned. However, the final result proved to be an innovative contribution to the field and has already been cited multiple times.

KEY CHALLENGES AND OPPORTUNITIES: MMPR Challenges Using an MMPR design presents some unique challenges for traditional academic and medical researchers. First, many researchers, and in particular clinical investigators, lack in-depth knowledge of phenomenology or qualitative methods in general. Phenomenological enquiry requires researchers to engage with unique methodological training in addition to engaging in the wealth of philosophical literature that informs such research approaches. Deductive enquiry based on the medical model is pervasive in health research, whereas inductive research aimed at understanding the subjective participant experience is unfamiliar, and integration of qualitative and quantitative methods is even less intuitive for those trained in traditional research methods. Therefore, for clinical investigators to engage with qualitative

Figure 18.1  Evolution of study design for illustrative case Author-created study described in Thompson et al., 2020.

ILLUSTRATING THE MIXED METHODS PHENOMENOLOGICAL APPROACH

methods, they must move away from the positivist paradigm and engage with alternative philosophical approaches, understanding the implications for knowledge, reality and research design. Further, MMPR designs require a unique combination of expertise and collaboration for the research team. Rigorous data-collection methods (e.g., in-depth interviews, transcription, reflexive journalling) and analytic processes (e.g., consensus/team coding, triangulation of data, member checking) demand more time than many more typical single-method studies. In-depth phenomenological interviewing requires time to build rapport and skilled interviewers who can elicit data-rich responses from participants who may not be accustomed to sharing their lived experience in a research setting. Finally, MMPR also provides great challenges in the presentation of method and findings. MMPR researchers must make challenging decisions regarding how to present the phases of their method and demonstrate integration and cohesion for publication. Furthermore, the complexity of the method, and volume of data mean that many academic journals that generally publish mono-method work (such as medical journals), present challenges for those submitting MMPR manuscripts, such as very limited word counts, strict regulations for numbers of tables and figures, and peer reviewers with limited understanding of rigour in qualitative methods. In a field where career advancement is dependent on numbers of publications, traditional methods may prove more enticing or accessible. This challenge could be aided by the development of universal reporting guidelines for mixed methods research similar to those existing for clinical trials (i.e. CONSORT) and epidemiological observation study designs (i.e. STROBE), as such tools increase the rigour, quality and respectability of research.

Opportunities Despite these recognized challenges, multiple benefits accompany the merging of paradigms, and it is critical to recognize the power of MMPR for advancing our understanding of important research questions involving the human experience. One strength of MMPR is the potential direct benefit to marginalized participant populations as MMPR utilizes multiple methodologies with an inductive focus rather than relying on assumptions about participants. Adolescents are one example of many populations that can benefit from this multifaceted approach. In this study, quantitative and qualitative results from adolescents and parents were not always congruent. Relying solely on standardized

263

questionnaires would have missed an important voice that was informed from the interviews, while at the same time categorization with quantitative data was necessary for full interpretation of the participant experience. Therefore, the complementary quan + phen approach was particularly appropriate as the qualitative results added context for interpreting the questionnaires, and the qualitative themes were enriched by integration of quantitative results. Similarly, other examples of MMPR approaches found in Table 18.1 have the potential to give a voice to other underrepresented groups. This study also illustrates that MMPR may be particularly useful in the study of health behaviours related to quality of life for individuals with rare and understudied disorders. Complicated health problems require an interdisciplinary clinical approach, with pragmatic guidelines for care across a variety of settings. Likewise, a diverse, interdisciplinary research team provides an excellent basis for successful MMPR. Research findings must be accessible to practitioners with diverse training backgrounds, and real-world implications of results must be apparent. The development of ecologically valid interventions and clinical care recommendations requires an in-depth understanding of patient priorities and lived experiences, best derived from a phenomenological strand of enquiry. However, more objective data contextualizing health outcomes and justifying the need for improved care is also necessary and is best collected through a more deductive strand. Integration of these methods has the potential to elucidate complicated health topics in a patientcentred and scientifically rigorous manner. Finally, MMPR lends itself to adaptation in multiple research fields. In addition to more traditional clinical-translational research, some MMPR methods may be particularly well suited for community-based participatory research, which relies on a partnership between researchers and the community. This field places great value in the lived experience of those in the community; therefore, phenomenology is a very natural fit. However, this can be greatly enriched when integrated with a complementary deductive approach as well. In a similar way, MMPR would be an excellent methodology in dissemination and implementation research (e.g. shared decision-making) and diversity, equity and inclusion initiatives.

CHAPTER SUMMARY This chapter advances the field of mixed methods research in providing an overview of mixed

264

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

methods phenomenological research presented alongside an illustrative example that demonstrates how this model can be practically applied in a manner that supports our commitment as researchers to equity, diversity and inclusion. While this model has been proposed in previous academic literature (Mayoh & Onwuegbuzie, 2015), this chapter is unique in providing an illustrative example of MMPR alongside an applied discussion to exemplify how these methods can be integrated within a single cohesive methodology to provide rich findings in a manner that supports commitment as researchers to equity, diversity and inclusion. We demonstrate how MMPR expands upon both phenomenological and quantitative methods and encourages these different approaches to inform each other. This advances the field of mixed methods research (MMR) in making a significant contribution to an emerging research narrative that focuses on the expansion of research designs that are rooted in one tradition (i.e. phenomenology) into a design that incorporates or interfaces with the other traditions. This chapter therefore expands practice by providing a methodological framework, and practical examples to researchers wishing to combine phenomenology with quantitative methods within a single study. In addition to providing an overview and review of the MMPR model, we also offer a justification for the compatibility of descriptive and interpretive forms of phenomenology with methods grounded in alternative paradigms, and summarize multiple models for MMPR research designs. The inclusion of the illustrative example demonstrates how research questions that require an understanding of lived experience to utilize traditional research methods may be particularly suited for an integrative MMPR approach. Using the current example, youth voices are often not adequately represented in clinical research or clinical practices. Therefore, the current design approach is a unique opportunity to capture these unique perspectives, as well as those of other underrepresented voices, rather than accepting prevalent assumptions. Our illustrative example demonstrates that MMPR is an ideal approach for tackling research questions where existing assumptions are based on facts from other areas and/or biological plausibility. MMPR provides researchers with a unique opportunity to select rigorous quantitative instruments based on this pre-existing knowledge and use phenomenology to both contextualize these results and inform new hypotheses. Finally, the illustrative example demonstrates how MMPR can be practically applied in a manner that supports a commitment to equity,

diversity and inclusion in providing rich livedexperiential data to accompany quantitative measures, thus including underrepresented, patient voices into the study, highlighting the subjective experiences and priorities of participants to form ecologically valid and pragmatic recommendations for clinical care. Based on the path shown by this chapter, we propose that in moving forward there is a need for researchers to provide further innovative examples of MMPR being utilized practically in alternative ways, and for different purposes (in line with the models presented in Table 18.1) to provide rich examples of MMPR to further strengthen the approach, and provide guidance and support for those wishing to adopt such methods to enrich their own research. In terms of professional practice, based on the review of the illustrative example and challenges faced, we suggest that there is a need for specialist training in both qualitative and mixed methods research within disciplines such as health sciences to provide researchers with the diverse skillset needed to conduct rigorous MMPR research that is both philosophically and practically sound. Furthermore, the development of reporting guidelines for mixed methods research (i.e. CONSORT, and STROBE) would help increase the rigour of MMPR approaches, and MMR more broadly. We advise all those considering MMPR to pay serious consideration to the cohesion between the integrated research methods, and the priority and sequencing of each methodological phase. Finally, we also encourage researchers utilizing MMPR to consider working within interdisciplinary, nonhierarchical research teams with topic knowledge from multiple perspectives to adequately construct qualitative themes and integrate qualitative and quantitative results.

WHAT TO READ NEXT Mayoh, J., & Onwuegbuzie, A. J. (2015). Toward a conceptualization of mixed methods phenomenological research. Journal of Mixed Methods Research, 9(1),  91–107.  https://doi.org/10.1177/1558 689813505358

This article is the original conceptualization paper for MMPR and provides an in-depth philosophical justification for using multiple forms of phenomenological method as components of mixed methods studies. It also provides numerous examples of MMPR in practice to underline potential models that can be used practically to inform the design of future research.

ILLUSTRATING THE MIXED METHODS PHENOMENOLOGICAL APPROACH

Mayoh, J., & Onwuegbuzie, A. J. (2014). Surveying the landscape of mixed methods phenomenological research. International Journal of Multiple Research Approaches, 8(1), 2-14. https://doi.org/ 10.5172/mra.2014.8.1.2

This paper explores the current implementation of MMPR within empirical research studies to provide a clear picture of how and why this research approach is being adopted. It also provides an overview of the key purposes for mixing phenomenology with methods grounded in alternative paradigms within a single study. Thompson, T., Zieba, B., Howell, S., Karakash, W., & Davis, S. (2020). A mixed methods study of physical activity and quality of life in adolescents with Turner syndrome. American Journal of Medical Genetics Part A, 182(2), 386–396. https://doi.org/ 10.1002/ajmg.a.61439

This is the full article from which the illustrative example within this chapter was taken. It demonstrates a practical example of MMPR being used to explore the quality of life of young women with Turner syndrome. The chosen design brought diverse, and often underrepresented, patient voices into the study, highlighting the subjective experiences and priorities of participants to form ecologically valid and pragmatic recommendations for clinical care. Furthermore, the MMPR approach required a team of investigators with diverse perspectives, training experiences and philosophies, resulting in a more inclusive and equitable research process.

REFERENCES Abrignani, M. G., Luca, F., Favilli, S., Benvenuto, M., Rao, C. M., Di Fusco, S. A., Gabrielli, D., & Gulizia, M. M. (2019). Lifestyles and cardiovascular prevention in childhood and adolescence. Pediatric Cardiology, 40(6), 1113–1125. https://doi.org/10.1007/ s00246-019-02152-w Amundson, E., Boman, U. W., Barrenas, M. L., Bryman, I., & Landin-Wilhelmsen, K. (2010). Impact of growth hormone therapy on quality of life in adults with turner syndrome. The Journal of Clinical Endocrinology & Metabolism, 95(3), 1355– 1359. https://doi.org/10.1210/jc.2009-1754 Biddle, S. J., & Asare, M. (2011). Physical activity and mental health in children and adolescents: a review of reviews. British Journal of Sports Medicine, 45(11), 886–895. https://doi.org/10.1136/ bjsports-2011-090185

265

Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage. Creswell, J. W., & Plano Clark, V.L. (2010). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage. Garza, G. (2007). Varieties of phenomenological research at the University of Dallas. Qualitative Research in Psychology, 4(4), 313–342. https:// doi.org/10.1080/14780880701551170 Giorgi, A. (2009). The descriptive phenomenological method in psychology: A modified Husserlian approach. Duquesne University Press. Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. The Annals of Family Medicine, 13(6), 554–561. https://doi/10.1370/ afm.1865 Hallett, C. (1995). Understanding the phenomenological approach to research. Nurse Researcher, 3(2), 55–65. https://doi/ 10.7748/nr.3.2.55.s6 Howe, K. (1988). Against the Quantitative-Qualitative Incompatibility Thesis or Dogmas Die Hard. Educational Researcher. https://doi/17. 10-16. 10.2307/1175845 Hutaff-Lee, C., Bennett, E., Howell, S., & Tartaglia, N. (2019). Clinical developmental, neuropsychological, and social-emotional features of Turner syndrome. American Journal of Medical Genetics: Seminars in Medical Genetics, 181(1), 126–134. https://doi.org/10.1002/ajmg.c.31687 Husserl, E. (1931). Ideas (W.R. Boyce Gibson, trans). George Allen & Unwin. Husserl, E. (1965). Philosophy as a rigorous science. In Phenomenology and the crisis of philosophy. New York: Harper and Row. Husserl, E. (2002). Philosophy as rigorous science. New Yearbook for Phenomenology and Phenomenological Philosophy, 2, 249–295. Johnson, R. B., McGowan, M. W., & Turner, L. A. (2010). Grounded theory in practice: Is it inherently a mixed method? Research in the Schools, 17(2), 65–78. Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14– 26. https://doi.org/10.3102/0013189X033007014 Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a Definition of Mixed Methods Research. Journal of Mixed Methods Research, 1(2), 112–133.  https://doi.org/10.1177/1558689806 298224 Kvale, S. (1996). Interviews: An introduction to qualitative research interviewing. Sage. Langridge, M. E., & Ahern, K. (2003). A case report on using mixed methods in qualitative research.

266

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Collegian, 10(4), 32–36. https://doi.org/10.1016/ S1322-7696(08)60074-8 Leech, N., & Onwuegbuzie, A. (2007). An array of qualitative analysis tools: A call for data analysis triangulation. School Psychology Quarterly, 22, 557–584. http://dx.doi.org/10.1037/1045-3830. 22.4.557 Mayoh, J., & Onwuegbuzie, A. J. (2015). Toward a conceptualization of mixed methods phenomenological research. Journal of Mixed Methods Research,  9(1), 91–107. https://doi.org/10.1177/ 1558689813505358 Morse, J. M. (2003). Principles of mixed methods and multi-method research design. In C. Teddlie, & A. Tashakkori (Eds.), Handbook of mixed methods in social and behavioral research (pp. 189–208). Thousand Oaks, CA: Sage. Robbins, B. D., & Vandree, K. (2009). The self-regulation of humor expression: A mixed method, phenomenological investigation of suppressed laughter. The Humanistic Psychologist, 37(1), 49–78. https:// doi.org/10.1080/08873260802394533 Sale, J. E., Lohfeld, L. H., & Brazil, K. (2002). Revisiting the quantitative-qualitative debate: Implications for mixed-methods research. Quality and Quantity, 36(1), 43–53. Schoemaker, M. J., Swerdlow, A. J., Higgins, C. D., Wright, A. F., Jacobs, P. A., & United Kingdom Clinical Cytogenetics, G. (2008). Mortality in women with Turner syndrome in Great Britain: A national cohort study. Journal of Clinical Endocrinology &

Metabolism, 93(12), 4735–4742. https://doi.org/ 10.1210/jc.2008-1049 Sienkiewicz-Dianzenza, E., Milde, K., & Frac, M. (2006). [Declared attitudes of girls with Turner’s syndrome towards physical education classes]. Endokrynol Diabetol Chor Przemiany Materii Wieku Rozw, 12(2), 124–126. https://doi.org/ 10.1023/A:1014301607592 Sienkiewicz-Dianzenza E, Milde K, Frac M. (2006). Lekcje wychowania fizycznego w opinii dziewczat z zespołem Turnera [Declared attitudes of girls with Turner’s syndrome towards physical education classes]. Pediatric Endocrinology, Diabetes and Metabolism. 12(2), 124–126. https://pubmed. ncbi.nlm.nih.gov/16813717/ Stawarska, B. (2009). Between you and I: Dialogical phenomenology. Ohio University Press. Thompson, T., Zieba, B., Howell, S., Karakash, W., & Davis, S. (2020). A mixed methods study of physical activity and quality of life in adolescents with Turner syndrome. American Journal of Medical Genetics Part A, 182(2), 386–396. https://doi. org/10.1002/ajmg.a.61439 Todres, L., & Holloway, I. (2004). Descriptive phenomenology: Life-world as evidence. In New qualitative methodologies in health and social care research (pp. 99–118). Routledge. Van Manen, M. (1990). Researching lived experience: Human science for an action sensitive pedagogy. University of New York Press.

19 Intersection of Mixed Methods and Case Study Research (MM+CSR): Two Design Options in Educational Research Loraine D. Cook and Vimala Judy Kamalodeen

INTRODUCTION There is a need for research that addresses and provides insights into the complexity of pressing social, economic, education and health problems. Existing research approaches such as case study (CSR) and mixed methods (MMR), have been used extensively across fields and disciplines as they are well suited to solving complex problems due to their adaptability and flexibility to multiple forms of data (Plano Clark et  al., 2018). However, under suitable circumstances, these two approaches may be intersected in a single study to offer unique methodological advantages for researchers to address the complexity of these research problems and issues (Plano Clark et al., 2018). According to Poth (2018), the complexity of research problems are “conditions in which the research system is integrated and yet too varied for understanding using simple, mechanistic and linear methods” (p. 304). Combining both approaches in investigating a complex phenomenon will facilitate the investigation to go outside the immediate boundary of the case, with the aim of tracking the issues and pursuing patterns of complexity to create a comprehensive understanding of the phenomenon (Poth, 2018; Stake, 1995).

We begin the chapter by providing a brief overview of case study research, including definition and relevant history. Next, we discuss the evolution of mixed methods case study research presenting two designs with relevant examples. Finally, we discuss a practical application of the combined approaches that highlights a single case study mixed methods design using an exploratory sequential design. This chapter helps to expand our thinking of the uses of MMR, in particular, case study mixed methods research, which enables us to examine more thoroughly complex research problems.

CASE STUDY RESEARCH (CSR) EXPLAINED What is Case Study Research? According to Gerring (2006), “case study is an intensive study of a single case or a small number of cases which draws on observational data and promises to shed light on a larger population of cases” (p. 28). A case study involves in-depth investigation within an everyday context or, as

268

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Yin (2014) stated, in a “real-world context”. It allows for the study of the complexity of real-life events. This strategy enables the researcher to examine different aspects of the case and how these various aspects of the case relate to each other (Ebneyamini & Moghadam, 2018). Case study research can be defined as “an ideal methodology when a holistic, in-depth investigation is needed” (Ebneyamini & Moghadam, 2018, p. 2).

History of Case Study The earliest published or formal use of case study research can be traced to Europe through the works of Bronislaw Malinowski and Frederic Le Play. Bronislaw Malinowski was a Polish-born Austrian who took refuge in Melanesia in the southwestern Pacific Ocean and lived for three years in the Trobriand Islands. There, Malinowski took the opportunity to do several observations of the local population and “what were to him its strange and exotic ways and customs” (Hamel et  al., 1993, p. 2). A case study from the Malinowski perspective involves intense and protracted observations of the particulars of a specific group. Particularizability is a critical characteristic of a qualitative case study (Butler-Kisber, 2010). Le Play (1806–1882) was a French researcher who systematically studied working-class populations across several European nations. Based on Le Play’s preliminary observations, he concluded that “that the family is this observation point or the case. An in-depth study of the family case would provide an understanding of society in its characteristic features. Therefore, he selected the working-class family as a unit” (Hamel et  al., 1993, p. 7). Under Le Play’s initiatives, 300 family monographs were produced. He was personally responsible for 100 of them. According to Hamel et al. (1993) “a precise and monographic study was conducted for each working-class family, using standardized approach” (p. 7). Le Play highlighted the importance of establishing a unit of analysis for case study research. Following the above, the Chicago School during the early 1900s to 1935—more specifically, The University of Chicago Department of Sociology— was prominent in the use of case study research (Tellis, 1997). According to Tellis (1997), the history of case study research is “marked by periods of intense use and periods of disuse” (p. 5). In the 1930s, the Chicago School was under pressure to identify with the scientific approach that “resulted in the denigration of case study as a methodology” (p. 5). The dispute evolved into the public domain in 1935 between professors from Columbia

University, who were championed quantitative researchers (the scientific method) and the Chicago School. The outcome was a continued decline in the use of case study research and the outcome was a victory for Columbia University. By the 1960s, however, researchers started to grapple with the limitations of the scientific methods. This evolved into a renewed interest in case study research. Strauss and Glaser’s development of the “grounded theory”, according to Tellis (1997), “along with some well-regarded studies accelerated the renewed use of the methodology” (p. 5).

Choosing the Case Choosing the case is sometimes referred to as case selection. Case selection can take two forms—a typical case or an exemplary case (an unusual or unique case) (Lichtman, 2013). A typical case could be identified by an expert in the field or the researcher conducting their field observations. Where the researcher takes the responsibility to determine the typical case, a set of criteria for guiding the selection of the typical must be developed. It is also helpful in deciding on the criteria for selection to include criteria for exclusion; in other words, what criteria are relevant and not relevant to the case. The typical case highlights the normal or average. Selecting an exemplary case is another approach recommended by Lichtman, referring to an outstanding or exceptional case. Such a case can be identified following data analysis of survey data. A scatter plot or box plot, for example, can be used to identify an exceptional case, called an outlier, using quantitative terms. The researcher can also rely on an expert to nominate that case for participation. The researcher may read accounts or have criteria to decide on the exceptional case. This type of case selection is unusual. This could also be a ground-breaking or creative case (Lichtman, 2013). Stake notes that the unusual case sometimes helps illustrate matters overlooked in typical cases. The selection of case(s) also depends on the purpose for the utilization of case study research. Stake (1995) categorized case studies as intrinsic or instrumental. We refer to a case study as intrinsic when the only purpose for doing the case is to understand the case itself. The case study is instrumental when we use it to understand something else (other than the case). Other authors such as Yin (2014) and Baxter and Jack (2008) categorized the case study’s purpose into explanatory, exploratory and descriptive. The purpose for doing an explanatory case study is to explain “how or why

INTERSECTION OF MIXED METHODS AND CASE STUDY RESEARCH

some conditions occurred or did not occur” (Yin, 2014, p. 238); we conduct an exploratory case study when our purpose is to “identify research questions or procedures to be used in a subsequent research study which might or might not be a case study” (Yin, 2014, p. 238); and a descriptive case study purpose “is to describe a phenomenon in a real-world context” (Yin, 2014, p. 238).

Establishing the Boundaries of the Case Establishing boundaries facilitates the development of the conceptual and methodological framework of the case study (Yin, 2014). Gall et  al. (2007) defined case study research as “the indepth study of one or more instances of a phenomenon in its real-life context that reflects the perspective of the participants involved in the phenomenon” (p. 447). A phenomenon can be a process, event, evaluation of a programme or any other issue of interest to the researcher. As noted in Gall et al.’s (2007) definition, a case “can be a particular instance in that phenomenon” (p. 447). It is important to note that the time and spatial boundaries of the case may change as the case study of those cases progress—”that is, cases may be re-cased” (Sandelowski, 2011, p. 155). Case study research is grounded in lived reality and allows us to examine complex interrelationships; the approach is flexible because it can be implemented at various points of the research process (Cook & Kamalodeen, 2020). Stake (2003) referred to case study research as having a “specific, unique, bounded system” (p. 136). It is important to establish the boundaries of a case—”a case is a specific, a complex, functioning thing” (Stake, 1995, p. 2). While a case may be bounded by time, place, event and activity, once the boundaries of the case have been established, it may be important for the researcher to search for additional data outside of the immediate scope of the case. This builds a context that is critical to interpreting the findings later on. A school, a classroom, a hospital or a unit within a hospital, an individual or a community of persons can form empirical boundaries for investigation.

Case Study Research Designs There are two types of single-case design: a single-case study with a holistic design or a case study with an embedded design. The holistic single case study has a single unit of analysis. Patton (2002) refers to the case as the unit of analysis, which could be individual people, clients

269

or a group of students. A case could be an individual, an organization or some issue. A single case with an embedded design may have more than one level. This occurs when a single case has a subunit or subunits of analysis (Yin, 2014), although for the holistic case study, Yin warns that as the researcher progresses through the study, different research questions may emerge from the initial one, and these dynamics are not unique to the case study. In addition, Stake encouraged researchers to be open to the evolution of the research questions within the holistic design. As the researcher gains greater understanding and new confidence, issues may be restated and the research questions changed. However, these dynamics in research do not negate Yin’s suggestion that one way to guard against “slippage” from the initial research questions is to have a set of subunits because this will increase sensitivity to change in the orientation of the initial research questions. When carrying out an embedded design, a researcher needs to guard against pursuing the subunits at the expense of the larger unit of analysis. On the other hand, a multiple case study is needed when the research investigation includes more than one single case study. The context of the study usually guides the choice of multiple case studies. A multiple or comparative case study allows the researcher to analyze within each context and across contexts (Baxter & Jack, 2008). Two advantages are associated with multiple case studies: (1) evidence produced from multiple case study design is more robust and reliable; and (2) multiple case studies produce support that is grounded in several empirical evidences, and so develops a more convincing theory (Baxter & Jack, 2008). The reason for pursuing a multiple case study is to identify and understand differences and similarities among cases (Baxter & Jack, 2008).

EVOLUTION OF MIXED METHODS CASE STUDY RESEARCH Mixing methods necessarily involves the integration of both qualitative and quantitative approaches, but where time, priority and dominance can vary (Leech & Onwuegbuzie, 2009). Although case study research is traditionally considered a qualitative strategy, there is no agreed set of methods for case study (Lincoln & Denzin, 2003; Luck et al., 2006). Hence, any set of methods from the various research approaches can help develop an understanding of a case. Thus, mixed

270

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

methods research provides a multiplicity of ways to collect, analyze and interpret data within a case for several years before the formal use of the label “mixed methods case study research”. While several authors have suggested that mixed methods case study research was formalized within recent times (e.g., Guetterman & Fetters, 2018; Yin, 2014), a historical journey examining studies that involved the joint use of quantitative approaches with qualitative case study reveals the use of qualitative and quantitative research approaches in case study research. A brief look at these historical uses provides illustrations of the intersection of qualitative and quantitative research approaches in case study research (e.g., Bonoma, 1985; Kaplan & Duchon, 1988). Bonoma (1985), in exploring the usefulness of case study research as an alternative method for marketers, discussed the general utility of combining quantitative and qualitative approaches in the research strategy. Bonoma noted that in the 1980s, there was a preference for quantitative research. However, he argued that qualitative is desirable “when a phenomenon is broad and complex, where the existing body of knowledge is insufficient to permit the posing of causal questions, and when a phenomenon cannot be studied outside the context in which it naturally occurs” (p. 207). Kaplan and Duchon (1988) combined quantitative and qualitative in case study research to understand the interrelationships between perceptions of work and a computer information system. After combining quantitative and qualitative methods, Kaplan and Duchon, at the end of their study, argued that the inconsistency of the results from an initial quantitative and qualitative data analysis was valuable because it allowed for further exploration of the research problem. Both researchers concluded that: Thus, triangulation of data from different sources can alert researchers to potential analytical errors and omissions. Mixing methods can also lead to new insights and modes of analysis that are unlikely to occur if one method is used alone. In the absence of qualitative data, the study would have concluded no reportable statistically significant findings. (p. 582)

In 2010, Onwuegbuzie and Leech used the label “mixed methods case study research to examine the issue of generalization in qualitative research articles published in selected journals. Later, Creswell and Plano Clark (2018), in their textbook Designing and Conducting Mixed Methods Research, discussed three complex mixed methods designs that included a mixed methods case study, also labelled as the mixed

methods comparative case study approach (p. 116). For a fulsome discussion of the evolutions in their thinking about designs, see also Chapter 2 (this volume). Guetterman and Fetters (2018) identified and characterized two methodological approaches to the intersection of mixed methods and case study research. These were the nested case study (mixed methods) and the nested mixed methods (case study). Finally, Cook and Kamalodeen (2020) described two approaches to intersecting CSR with MMR. When researchers start the investigation with case study research and embed mixed methods within it, it is called “case study mixed methods research” (CS-MMR). Or when the investigation begins with mixed methods research and cases are generated thereafter based on initial data analyses, the design is called “mixed methods case study research” (MM-CSR). The nomenclature is specific to the approach that initiates the design. This present chapter draws upon our experiences with CS-MMR.

INTERSECTION OF MMR AND CASE STUDY RESEARCH (MM+CSR) Given the complexity of research problems, we need to be flexible and adaptable in our research thinking. Hence, MM researchers should be less rigid in implementing their research plans in a given research context, especially when expected conditions of the original research design change. In embracing the intersection of CSR with MMR, researchers can expand the conversation on utilizing what Creswell and Plano Clark (2018) referred to as “complex applications of mixed methods design”. As such, intersecting MMR and CSR allows for another way of enquiry that harnesses the commonalities of both research approaches. Such an intersection allows for the components of the research process to work together to give a more holistic understanding of the research issue. We call this intersection “MM+CSR” to denote the combined approaches broadly. Although case study research offers a “bridge across traditional research paradigms” (Luck et  al., 2006, p. 103), there are challenges when intersecting CSR with MMR. As a stand-alone, CSR generates a massive amount of data. When adding MMR to the design, researchers need to be cognizant of managing the collection and analysis of large data. Hence, the researcher needs to make decisions based on the rationale and purpose for intersecting MMR and CSR. To complete the study, the researcher will also need to manage the

INTERSECTION OF MIXED METHODS AND CASE STUDY RESEARCH

time span and the available resources (for example, research assistants and costs associated). Another challenge when combining CSR with MMR is researchers’ limited expertise in multiple approaches and methods required to investigate using this complex approach. For example, a researcher needs knowledge and skills in quantitative and qualitative approaches and methods. In analyzing the data, for example, the skills for narrative writing in the qualitative phase

271

are needed to highlight participants’ voices, whereas statistical techniques are needed for analysis of the quantitative data, which can be quite involved. Several authors (e.g., Plano Clark et al., 2018) suggest collaborative research where experts from the different research approaches (quant and qual) team up to provide a broader range of skills and experiences to a particular study. While comparing case study and mixed methods research, Table 19.1 summarizes the

Table 19.1  A comparison of key attributes of MMR and CSR and the commonality between MMR and CSR

Core principles

Paradigm

Rationales

Challenges

Mixed methods research

Case study research

Intersection of MMR and case study

Integration refers to how one brings together the qualitative and quantitative results in a mixed methods study. Mixed methods research encourages the use of multiple worldviews.

An integrative system: a set of parts working together to make a whole.

Intersection is the interface of MMR and case study research for holistic understanding.

Case study research uses a wide variety of research methods and allows for multiple worldviews. Instrumental We do the case study because the case will help the researcher to understand something else other than that particular case.

The intersection allows for multiple worldviews.

Rationales include triangulation, complementarity, expansion and enhancement, etc.

Qualitative case study research can be used to develop a quantitative survey to expand on the QUAL findings. Mixed methods can be used within case study research for intrinsic reasons.

Intrinsic We have an intrinsic interest in the case; the researcher wants to understand the case only. Researcher skills Researchers There is too much data for MMCSR researcher needs to may have knowledge and easy analysis have knowledge and skills in skills in only qualitative or quantitative and qualitative The case study can take too long quantitative approaches, approaches and methods: the and can result in massive whereas both are required for skills for narrative writing in amounts of data. a mixed methods study. a case study to highlight a participant voice, as well as Oversimplification or Limits in time and skills in statistical analysis. exaggeration of a resources situation Researchers should keep in mind Merriam (1998) in her seminal that qualitative phases can publication, stated that case take longer than quantitative studies could oversimplify or ones, so MMR studies can exaggerate findings. take longer than when one approach is used. Researcher’s bias The need to educate others about MMR Not all scholars have embraced mixed methods as an approach.

Source: From Cook and Kamalodeen, 2020, p. 57.

Researchers need to guard against the overemphasis on some segments of the data because of researcher’s preference.

272

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

possible challenges associated with combining CSR and MMR.

Mixed Methods Case Study Research (MM+CSR) Designs Mixed methods case study research (MM+CSR) is the intersection of mixed methods and case study research. The purposes of conducting mixed methods case study research are similar to those of mixed methods research. For example, Johnson and Christensen (2000) identified the following purposes of MMR: (1) to produce a more comprehensive understanding of a research problem; (2) to add contextualized understanding to quantitative findings; (3) to provide evidence of both descriptive and explanatory/process causations; (4) to obtain a deeper and more complex understanding of statistical findings; (5) to obtain multiple perspectives about the research phenomenon; and (6) to provide categories and constructs to be tested in the quantitative research phase. Overall, a mixed methods case study researcher is desirous of having a more complex understanding drawing from multiple perspectives about the research phenomenon being studied.

There are two basic mixed methods case study research designs purported by Guetterman and Fetters (2018), and Cook and Kamalodeen (2020). They are mixed methods case study research and case study mixed methods research, already described. The following section describes each design with an example each. In case study mixed methods research (CS-MMR), the researcher decides from the outset to conduct case study research that incorporates a mixed methods research design. For example, the case study could include a survey of participants within the case (Cook & Kamalodeen, 2020; Yin, 2014). The case study could incorporate an existing mixed methods design such as exploratory sequential, explanatory sequential or convergent or even intervention (Creswell & Plano Clark, 2018). Figure 19.1 illustrates a CS-MMR design where the case of game-based learning (GBL) in STEM primary level classrooms utilized a MM intervention design to collect both quantitative and qualitative data. For a discussion of an additional example of game-based research integrations, see also Chapter 24 (this volume). In a mixed methods case study research design (MM-CSR), the researcher decides to conduct a mixed methods study from the onset. This usually involves an explanatory design where a survey is

Case: Game-based learning in selected STEM primary schools with embedded interven�on MMR design

BEFORE Quan�ta�ve data –

INTERVENTION-GBL GBLGBLGBL

student pre test

AFTER Quan�ta�ve data – student post test

Qualita�ve Data –

DURING

Teacher ques�onnaire on a�tudes to GBL and geometry

Qualita�ve data – Teacher observa�ons of students engagement with games

Qualita�ve Data – Teacher interview of GBL experience

Quan�ta�ve data – Student ‘rate my game’ scales

Figure 19.1  Illustration of a CS-MMR embedding an MM intervention design in a gamebased learning investigation Source: Cook & Kamalodeen, 2020, p. 63.

273

INTERSECTION OF MIXED METHODS AND CASE STUDY RESEARCH

undertaken followed by quantitative data analysis; thereafter, the researcher(s) identifies a case or cases for more in-depth case study of the research issue using a qualitative research approach (Cook & Kamalodeen, 2020; Yin, 2014). The findings from the qualitative data analysis are then used to expand and give insights into the quantitative results. Figure 19.2 illustrates the four cases for the qualitative investigation that emerged from the quantitative data analysis of teachers.

DESCRIPTION OF AN APPLICATION OF CASE STUDY MIXED METHODS RESEARCH (CS-MMR) This application of case study mixed methods research involved faculty members from three campuses of a five-campus university. The faculty members’ experiences were examined by the end of the first COVID-19 academic year (2020/21). All faculty members are teacher educators at the Schools of Education of three selected campuses represented in the study.

Type of Case Study We used Grandy (2010) as a guide to identify the dominant purpose of the case, which is intrinsic, generally referred to as an “intrinsic case study”.

Grandy (2010) stated that “the intrinsic case is often exploratory in nature, and the researcher is guided by her interest in the case itself rather than in extending theory or generalizing across cases” (p. 473). This case study focused on exploring the phenomenon, in-depth, of teacher educators working at home during the pandemic—a phenomenon still being experienced at the time of writing this chapter. We did not intend to test a theory or apply an intervention. We were less interested in generalizing findings to other cases (Grandy, 2010; Stake, 1999). This particularity is an essential attribute of the case study (Thomas, 2010). In other words, an intrinsic case study focuses on the case itself. The intrinsic study could provide insights into teacher educators’ experiences during stressful and uncertain times and lead to further research.

Design We have purposefully selected a CS-MMR study to illustrate key aspects of the intersection of case study and mixed methods research for this chapter. We chose an exploratory sequential design where the qualitative phase initiated the study. The intent of integration was to “build to a survey instrument specific to a particular group (Creswell & Poth, 2017), such as teacher educators at the university. However, similar to other instruments, this instrument can be adapted to other similar contexts. The main idea was to incorporate selected participants’ views in building the instrument and

MIxed Methods Study with four emergent qualita�ve cases

QUANTITATIVE

Case 1 Teacher 1

Case 2

Case 3

Case 4

Teacher 2

Teacher 3

Teacher 4

Figure 19.2  Illustration of a MM-CSR explanatory sequential design Source: Cook & Kamalodeen, 2020, p. 68.

274

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

ensuring the particularities of the case are addressed and explored among a larger group of teacher educators (see Figure 19.3). This contextspecific instrument brings the advantage of fleshing out the nuances and complexities of the phenomenon being studied. It is important to note that CS-MMR is not restricted to studies that aim to build an instrument, nor does instrument development require case study methods.

INTEGRATING THE QUAL AND QUAN IN AN EXPLORATORY SEQUENTIAL CASE STUDY DESIGN Integrating the qual and quan data allowed for a comprehensive understanding of the phenomenon within the boundary of the case. The exploratory sequential design allowed for building an instrument relevant to the phenomenon being studied using the narrow base of six faculty members and the two researchers’ reflections (also faculty members). In contrast, the quan allowed the researchers to broaden the number of participants within the case context. Case study mixed methods research (CS-MMR) allowed us to gain a comprehensive understanding

of the experiences of teacher educators in the new virtual working environment caused by COVID-19. Integrating the qual and quant allowed us to access a wide range of actions and responses of teacher educators under the prevailing circumstances that were not previously explored within the context of the case. For example, using the narrative from the qualitative data, we were able to describe how and why the educators worked efficiently at home in the pandemic; while the quant gave a numeric value as to the extent to which educators had similar experiences as our interviewees. The quant showed that more than 50 per cent strongly agreed that the participants perceived themselves as able to do remote work efficiently. As pointed out earlier, the limitation of this single mixed methods case study is that we cannot generalize the results from this study to other faculties across the campuses. However, the survey instrument can be adapted and used by other Caribbean researchers.

DISCUSSION AND IMPLICATIONS OF USING CS-MMR IN THE PANDEMIC ERA Case study mixed methods research design with an exploratory sequential mixed methods process

Figure 19.3  Application of case study mixed methods (CS-MMR) using an exploratory sequential design

INTERSECTION OF MIXED METHODS AND CASE STUDY RESEARCH

provided us with a methodological framework to investigate the pandemic era caused by COVID19. This was considered appropriate as our research phenomenon during COVID-19 had no precedence to inform our methods, especially our data-collection instruments. COVID-19 presented a new dynamic and uncertain research environment with little preliminary data available for conducting investigations. The case study research with exploratory sequential MM allowed us to understand the realities of the teacher educators experiencing a new work context that was suddenly thrust upon them. CS-MMR allowed for this empirically driven investigation of faculty experiences of remote work during the pandemic through meaningful integration of research approaches. In addition, the intrinsic purpose for doing case study mixed methods research allowed us to have a deeper and broader understanding of this particular phenomenon being investigated, starting with a small group of teacher educators, then expanding to the larger group within the case. COVID-19 also presented opportunities for us to adapt our research methods to the new online environment. For example, case study researchers usually study the natural research settings of the investigation over some time. Researchers who are external to the study context cannot experience the research context easily unless some virtual ethnography is involved. However, we overcame this constraint because, as researchers, we worked in the same context as our participants before and during COVID-19. Hence, the online researcher will need to prepare innovative ways of experiencing and understanding the context of the case being studied through the eyes of their participants and yet somewhat independent of them. Perhaps collecting detailed descriptions of the context of the case, with photographs, graphics and videos, can provide credible data about the context of the case. Furthermore, debriefing can be useful in obtaining further contextual information of the case. During post-interview debriefing, clarifying questions can be asked to address gaps or errors in the transcript’s content. In so doing, the researcher can obtain further knowledge about the context that will give a more in-depth understanding of the previous interview (Collins et al., 2013). Using a combined strategy of case study and mixed methods research requires detailed planning of the case and the embedded mixed methods design. In addition, the data collection is both intense and extensive, drawing on a range of skills. Even as a single case study, this combination can be demanding and burdensome on one researcher due to the vast amount of qualitative data to be collected and analyzed. We found that the team approach to research

275

helped to reduce this burden, and interactions enhanced the experience, especially when there is mutual respect in the research team. The lively discussions allowed the freedom to disagree and work through the disagreement, guided by the literature, thus leading to knowledge building and knowledge sharing. Consulting the growing body of texts, materials and workshops available to MMR researchers can be fulfilling. As team researchers, we were able to search, locate and share literature that allowed us to create a rich repository of literature that further allowed us to make sound and rational judgements; without the team approach, this would have been challenging. Thus, the team approach leveraged affordances of information sharing that would not be possible individually. The combined approaches of case study and mixed methods research in understanding a single case allowed for depth and breadth of the investigation, especially in cases with large research populations. In our case, even though it was bound to teacher educators at one university, this involved recruiting participants across three campuses. Additionally, the built survey instrument allowed the researcher to have a greater capacity to gain a more complete and nuanced understanding of the case, rather than utilizing one research approach only.

CONCLUSION This chapter expands how we think about intersecting mixed methods research with other research approaches like case study. MM+CSR draws upon our collective knowledge and experiences as quantitative, qualitative and mixed methods researchers. Even as a single case study, the MM+CSR combination can be demanding and burdensome on one researcher due to the vast amount of data to be collected and analyzed. We recommend a team approach to enhance the quality of the research process and completion of the study in a timely manner; teamwork helps to address the gaps in knowledge and experience in an individual researcher. The intersection of case study and mixed methods research requires thorough planning of the case and the embedded mixed methods design. For example, MM+CSR involves intense and extensive data collection in that a researcher needs to be knowledgeable of the data collection techniques for both quantitative and qualitative research. In addition, in MM +CSR the researcher must also learn to nest real-world events within the needs of the data-collection process (Yin, 2014).

276

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Before embarking on the findings and analysis of the case study research aspect of the MM +CSR, it is important for the researcher(s) to justify selecting a case study as an appropriate frame for the study and clarify why the mixed design is appropriate (Thomas, 2016). This chapter contributes to the mixed methods community of practice in describing the intersection of mixed methods and case study research (MM+CSR). To understand the intersection, we discussed the two approaches (CSR) and (MMR) separately, highlighting critical attributes that allow the researchers to understand each approach individually. We expand understandings of this intersection through a detailed discussion of the two MM+CSR designs: case study mixed methods research (CS-MMR) and mixed methods case study research (MM-CSR) using relevant examples. We further detailed a practical application of CS-MMR by discussing the processes and procedures used by the researchers in examining faculty members’ experiences with remote work during the pandemic era. We highlighted challenges with the approach and how we responded to methodological challenges. Research approaches such as case study and mixed methods with an exploratory sequential design can be used across disciplinary applications due to their adaptability and flexibility in facilitating research in novel contexts. Finally, the chapter provides guidance and insights in designing and investigating a phenomenon using a mixed methods case study.

Grandy discussed the key attributes of an instrumental case study research. The author used examples to illustrate the application of instrumental case study in influencing the purpose of the case being studied. An important feature in the discussion is that in planning a case study for instrumental purposes the researcher should identify similar cases to the case being studied. Grandy noted, “In this way, the researcher will use the instrumental case to explore in depth a particular phenomenon and then compare this case with other cases, so that the reader can see the transferability of the case findings” (p. 474).

WHAT TO READ NEXT

REFERENCES

Guetterman, T. C., & Fetters, M. D. (2018). Two methodological approaches to the integration of mixed methods and case study designs: A systematic review. American Behavioral Scientist, 62(7), 900–918. https:// doi.org/10.1177%2F0002764218772641

Guetterman and Fetters formalized the nomenclature of the two approaches to combining mixed methods and case study research. They presented these two designs with useful illustrations. They presented a review of the literature integrating mixed methods and case study designs and described key methodological features and discuss four exemplar interdisciplinary studies. Grandy, G. (2010). Instrumental Case Study. In A.J. Mills, G. Eurepos & E. Wiebe (Eds.), Encyclopaedia of case study research, (Volumes 1&2, pp. 473–475). Sage. http://dx.doi.org/10.4135/9781412957397

• Luck, L., Jackson, D., & Usher, K. (2006). Case study: a bridge across the paradigms.Nursing Inquiry, 13(2), 103–109. https://doi.org/10.1111/j.1440-1800. 2006.00309.x While several prominent authors have discussed case study research, these authors offer the notion of case study as “a bridge across the traditional research paradigms” with broad research application and epistemological, ontological and methodological flexibility. This idea presents a conceptual frame for researchers wishing to combine case study research with other approaches such as mixed methods. This fluidity is important and, as case study has been traditionally presented as a stand-alone qualitative approach, adds support for methodological discussions of qualitative case study research.

Baxter, P., & Jack, S. (2008). Qualitative case study methodology: Study design and implementation for novice researchers. The Qualitative Report, 13(4), 544–556. https://doi.org/10.46743/2160-3715/ 2008.1573 Bonoma, T. V. (1985). Case research in marketing: Opportunities, problems, and a process. Journal of marketing research, 22(2), 199–208. https://doi. org/10.1177/002224378502200209 Butler-Kisber, L. (2010). Qualitative inquiry: Thematic, narrative and arts-informed perspectives. Sage. Collins, K. M., Onwuegbuzie, A. J., Johnson, R. B., & Frels, R. K. (2013). Practice note: Using debriefing interviews to promote authenticity and transparency in mixed research. International Journal of Multiple Research Approaches, 7(2), 271–284. https://doi.org/10.5172/mra.2013.7.2.271 Cook, L. D. & Kamalodeen, V. J. (2020). Combining mixed methods and case study research (MM+CSR)

INTERSECTION OF MIXED METHODS AND CASE STUDY RESEARCH

to give mixed methods case study designs. Caribbean Journal of Mixed Methods Research, CJMMR, 1(1), 47–76. UWI Press. www.uwipress.com/ journals/the-caribbean-journal Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research. (3rd ed.). Sage. Creswell, J. W., & Poth, C. N. (2017). Qualitative inquiry and research design: Choosing among five approaches. [kindle] Sage. Ebneyamini, S. and Mohammad Moghadam, R. S. (2018). Toward developing a framework for conducting case study research, International Journal of Qualitative Methods, 17, 1–11. https://doi.org/ 10.1177%2F1609406918817954 Gall, M. D., Gall, J. P., & Borg, W.R. (2007). Educational Research. Allyn & Bacon. Gerring, J. (2006). Case study research: Principles and practices. Cambridge University Press. https://doi. org/10.1017/CBO9780511803123 Grandy, G. (2010) Instrumental case study. In A. J. Mills, G. Eurepos & E. Wiebe (Eds.), Encyclopaedia of case study research (Vols. 1 & 2, pp. 473–475). Sage. Guetterman, T. C., & Fetters, M. D. (2018). Two methodological approaches to the integration of mixed methods and case study designs: A systematic review. American Behavioral Scientist, 62(7), 900–918. https://doi.org/10.1177%2F000276421 8772641 Hamel, J., Dufour, S., & Fortin, D. (1993). Case study methods. Sage. https://doi.org/10.4135/97814129 83587 Johnson, B., & Christensen, L. (2000). Educational research: Quantitative and qualitative approaches. Allyn & Bacon. Kaplan, B., & Duchon, D. (1988). Combining qualitative and quantitative methods in information systems research: A case study. MIS Quarterly, December, 571–586. Leech, N. L., & Onwuegbuzie, A. J. (2009). A typology of mixed methods research designs. Quality & Quantity, 43(2), 265–275. https://psycnet.apa.org/ doi/10.1007/s11135-007-9105-3

277

Lichtman, M. (2006). Qualitative research in education: A user’s guide. Sage. Lincoln, Y. S., & Denzin, N. K. (Eds.). (2003). Turning points in qualitative research: Tying knots in a handkerchief (Vol. 2). AltaMira Press. Luck, L., Jackson, D., & Usher, K. (2006). Case study: a bridge across the paradigms. Nursing inquiry, 13(2), 103–109. https://doi.org/10.1111/j.14401800.2006.00309.x Merriam S.B. (1998) Qualitative Research and Case Study Applications in Education. Jossey-Bass Publishers, San Francisco. Onwuegbuzie, A. J., & Leech, N. L. (2010). Generalization practices in qualitative research: A mixed methods case study. Quality & Quantity, 44(5), 881–892.  https://doi.org/10.1007/s11135-0099241-z Patton, M. (2002). Qualitative research and evaluation methods (3rd ed.). Sage. Plano Clark, V. L., Foote, L. A., & Walton, J. B. (2018). Intersecting mixed methods and case study research: Design possibilities and challenges. International Journal of Multiple Research Approaches, 10(1), 14–29. https://doi.org/10.29034/ijmra. v10n1a1 Poth, C. N. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Sandelowski, M. (2011). ‘‘Casing’’ the research case study. Research in Nursing & Health, 34, 153–159. https://doi.org/10.1002/nur.20421 Stake, R. E. (1995). The art of case study research. Sage. Stake, R.E (2003). Case study. In N. K. Denzin & Y.S. Lincoln (Eds.). Strategies of qualitative inquiry (pp. 134–164). Sage. Tellis, W. M. (1997). Introduction to case study. The Qualitative Report, 3(2), 1–14. https://doi. org/10.46743/2160-3715/1997.2024 Thomas, G. (2016). How to do your case study. Sage. https://us.sagepub.com/en-us/nam/howto-do-your-case-study/book242255 Yin, R. (2014). Case study research: Design and methods. Sage.

20 Harnessing Mixed Methods for Research Instrument Development and Legitimation Va n e s s a S c h e r m a n a n d L i s a Z i m m e r m a n

INTRODUCTION This chapter explains how instrument development from quantitative or qualitative perspectives can be combined meaningfully to account for the complexity in mixed methods design and data integration. We describe a mapping instrument tool for guiding the integration to enhance the legitimation of the mixed methods research inferences made as well as the validity, reliability and trustworthiness from a single (i.e., qualitative or quantitative) design perspective. This work is important to further the use of mixed methods research in instrument design. Careful design and development are even more pertinent today, given the cultural and social diversity that social scientists engage with daily and the need to account for context (Nastasi & Hitchcock, 2016), and the relevance of research for both participants and the intended audience of the research. For a discussion about indigenous cultural values instrument development using mixed methods, see also Chapter 14 (this volume). Mixed methods research has been under development for some time. Scholars are constantly engaging with the field to think through methodological issues and what Mertens (2018a) calls wicked problems. A critical methodological issue and a wicked problem would be satisfying legitimation claims, which refers to the validity and quality criteria in mixed methods studies. Validity and the applicability thereof have

a long tradition in psychology and psychometrics. Messick (1993), when considering validity implications, also reflected on the social impact and/or consequences of instrument use. The reflections may have been in recognition of psychometric media and care in the development and use thereof, but the sentiment can be extended to all research fields. In this chapter, we first orientate the reader in terms of how instrument development takes place from quantitative or qualitative perspectives and how these can be combined meaningfully to account for the complexity in design and data integration. We thus first consider single instrument design from both quantitative and qualitative approaches. Thereafter, we discuss the landscape status quo with mixed methods instrument development designs. This is followed by a reflection on legitimation considerations for extended designs (methodological norms, qualitative, quantitative and mixed methods). The chapter concludes with some suggestions for further reading.

ORIENTATION TO THE IMPORTANCE OF VALIDITY IN RESEARCH OUTCOMES Messick (1993) views validity as the appropriateness, meaningfulness and usefulness of inferences drawn as it applies to qualitative and quantitative instruments. In consideration of this, Scherman

HARNESSING MIXED METHODS FOR RESEARCH INSTRUMENT DEVELOPMENT

and Liebenberg (2021) have proposed that a mixed methods research approach should be followed when considering instrument development to establish psychometric and cultural value. One of the strengths of mixed methods is to augment the perceived weaknesses of any single research approach, and the same can be said about instrument development within mixed methods research. In this regard, Poth (2018) speaks to the complexity that comes with the territory of implementing a mixed methods study. While researchers may seek simple solutions in the form of designated designs to implement a mixed methods research undertaking, the reality of the implementation may lead to numerous questions regarding the actual practical process. Many applied studies outline using data from one approach to inform the development of a scale questionnaire (as examples, see Bearss et al., 2016; Connell et al., 2018; Fredericks et al., 2016; Lee et al., 2015), but few deal in detail with actual practical processes and considerations aligned to implementation, nor do they, in some cases, overtly recognize the involvement of a mixed methods approach. In general, researchers reporting instrument development studies may not be cognizant of their utilization of a mixed methods approach to instrument development, which highlights the need to bring its use to the fore in the methodological literature. While the pressing need for practical guidance endures, some notable efforts have made important contributions (i.e., Daigneault & Jacob, 2014; David et  al., 2018; Koskey et  al. 2018; Mertens, 2018b; Onwuegbuzie et al. 2010; Younas et al. 2020). Daigneault and Jacob (2014) specifically assert that we need more contributions about why and how to mix methods for validation purposes in terms of instrument development. There is thus room for illustrating in detail how mixed methods can be optimally used for instrument development. Therefore, how can qualitative data analysis techniques, such as constant comparison, key words in context, word count, content analysis and taxonomic analysis, be used to develop questionnaires? Likewise, how can questionnaires be analyzed in different ways? This would mean drawing on parametric and non-parametric statistics where required, or making use of Rasch analysis or Exploratory and Confirmatory Factor Analysis to develop qualitative protocols. The argument can also be extended to establish methodological norms, how construct-related validity can be explored within the framework of qualitative data, and how quantitative data can be analyzed from a qualitative perspective to ensure that validity and trustworthiness can be empirically established.

279

The argument can be extended even further, considering data integration, whether quantitative monoanalysis, qualitative monoanalysis or mixed analysis (Onwuegbuzie & Johnson, 2021). If integration across qualitative and quantitative research outcomes is considered a gold standard for an excellent mixed methods study, researchers will inevitably interrogate the meaning of such integration and when this must occur. In our experiences with mixed methods both as researchers and postgraduate supervisors, the actual integration is often dealt with in the comparison, juxtaposition and merging of findings for both the qualitative and quantitative components of a study. However, suppose that we are to carefully design mixed methods studies as we do a single approach study. In that case, we need to look at how our two approaches speak to each other, from the initial conceptualizations to the conceptual and/or theoretical framework to the research questions and aims, the design choices and the methods. This so-called chain of reasoning (Krathwohl, 1998) in the research, where the methodological and conceptual choices at each stage logically link to each other, may seem obvious. Still, this is not always clear for novice researchers, especially in light of the iterative nature between many of the stages. We know this as we grappled with the same understandings as new researchers. From such a perspective, it makes sense for a mixed methods study that, where appropriate, the instruments for each approach speak to each other in a way the research process/findings from one approach feed into the development of data-collection foci for the other approach. By data-collection foci, we, of course, mean instrument design and the need to justify our choices (Shannon-Baker, 2018). In this way, if we look more closely at integration in any mixed methods study and not only research with the purpose of validating a research instrument for further use, then we should carefully consider the instruments that are used for both quantitative and qualitative components of the research, and how these speak to each other with the goal of data integration and validation of research participant world views.

SINGLE METHODOLOGY INSTRUMENT DESIGN Invariably, in considering the use of mixed methods for enhanced research instrument development, the practices used for instrument development from both quantitative and qualitative approaches as singular processes must be scrutinized. In this section, we review the practices and processes for

280

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

the development of quantitative and qualitative instruments.

The Development of Quantitative Instruments Regardless of which research textbook you review, there will be a section on questionnaire development as part of survey research. The aesthetics of the product are addressed as instruments and should be attractive, brief and easy to respond to, must not include items that do not relate to the topic, and preferably be structured (Gay et  al., 2012). What is generally missing from the literature on research design related methods is the use of questionnaire frameworks to plan the content to fulfil content-related validity requirements. Questionnaire frameworks are similar to test specifications, where test specifications can be understood as an organizing tool in which content domains are mapped to outcomes and taxonomies such as Bloom’s taxonomy (Anderson et  al., 2001), the difficulty of items, and weighting of domains. This is the initial step before proceeding to write the items. Much in the same manner, a questionnaire framework maps the sections of the questionnaire in terms of the construct or attribute being tested, the definition applicable, number of items, item type, coding relevant and possible analysis. The questionnaire framework can then be related to the conceptual framework for the study and the research questions. Using a framework such as this strengthens the chain of reasoning that we keep coming back to. Table 20.1 provides an example of such a framework. Once the questionnaire framework is completed and there is a clear understanding of the construct and its relation to the conceptual framework, item writing can commence. Figure 20.1 provides an

overview of the various considerations concerning item construction (Gay et al., 2012). In the planning of the research, it is also suggested that the instrument be piloted not only to identify the strengths and the weaknesses, but also to adhere to validity requirements. Validity is generally an umbrella term in which content-related validity, construct validity and criterion-related validity are subsumed. Validity is thus multifaceted and evidence on several facets is required to substantiate claims. For example, when exploring the coverage of items across a domain (the exploration of content-related validity), then considerations regarding the prediction of future achievement (investigating criterion-related evidence, predictive validity) would not be included in the interpretation. Content-related validity evaluates whether the instrument consists of items that sufficiently cover the content domain (Murphy & Davidshofer, 1994). Content-related validity is normally evaluated by constructing a table of specifications (Suen, 1990). In terms of the field of Psychology, evidence for construct-related validity reflects whether an instrument is illustrative of psychological characteristics or underlying traits embedded in the theoretical construct of interest, and whether the theory behind the instrument is supported by the results (Gronlund 1998; Scherman, 2016; Suen, 1990). Construct-related validity thus links to construct modelling, which provides a framework for instrument development, drawing our attention to the purpose of the instrument being developed and the context in which the instrument will be used (Wilson, 2005). Exploring a construct from an empirical perspective can be viewed from two perspectives. The first is Classical Test Theory (CTT), where we would undertake a factor analysis. The second is from a modern test theory approach using Item Response Theory (IRT) or Rasch analysis. Factor analysis is considered an

Table 20.1  An example of a questionnaire framework for quantitative instruments Construct

Definition

Number of items Item type

Professional Vocational training 26 Items development/ encourages the improving further development practice of staff (Sammons, 1999) as articulated by in-service training opportunities, updating policies and introducing new programmes (Taggart & Sammons, 1999). Source: Author created.

dichotomous items. Likert scale items.

Coding

Possible analysis

No = 1 Yes = 2 4 = Strongly agree 3 = Agree 2 = Disagree 1= Strongly disagree

Frequencies. Cross-tabulations. Scale analysis (including reliability analysis). Correlations.

HARNESSING MIXED METHODS FOR RESEARCH INSTRUMENT DEVELOPMENT

281

Only include items which relate to the topic Be specific – short simple items are best

Collect demographic information

Item construction

Avoid leading and double-barrel questions

Avoid sensitive or touchy questions

Each question should deal with a single concept

Avoid jargon

Figure 20.1  Guidelines related to item construction Source: Author created.

exploratory method, which guides hypotheses or informs researchers (and practitioners) about the underlying patterns in data sets (Field, 2017). At the heart of IRT is the principle that there is a relationship between an individual’s ability to endorse a response option and how they respond to the item (Tran et  al., 2017). In this way, IRT explores instruments on an item level, instead of a test level, to characterize individual respondents (termed “latent traits”) to identify the probability of a positive response (Hambleton et  al., 1991). The Rasch model, which is another method under the banner of modern test theory, contributes to inferences made about construct validity and indicates how well the item fits within the underlying construct (Bond & Fox, 2015). The construct validity-related techniques are ideal for exploratory data analysis where we want to understand the structure of items or identify those items that are functioning well. Of course, both approaches are relevant in mixed methods designs, as illustrated by Onwuegbuzie et al. (2010) and Scherman and Liebenberg (2021).

The Development of Qualitative Instruments Despite an inductive logic inherent in the constructivist and interpretivist foundations of qualitative research, qualitative studies begin with preconceived knowledge, ideas, concepts and theories which drive data collection and ultimately mould the associated instruments. Qualitative instruments (usually referred to as research protocols, schedules or guides, but could also include other media and projective techniques) take many forms, with the most prominent being observation and individual or focus group interview schedules. The development of the instruments follows similar processes. The research interview is the dominant datacollection tool in qualitative research, and it is our focus for our purposes of illustration in this chapter as the proposed guidelines can be extrapolated to other data-collection instruments. The aim of a qualitative research interview is to obtain nuanced descriptions of the interviewee’s interpretation of the phenomenon under investigation (Kvale,

282

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

1996). Qualitative interviewing is thus a directed conversation which permits in-depth exploration of a particular topic from the experience of the participant (Charmaz, 2006). The formulation of a qualitative interview schedule is aligned to specific research contexts, disciplinary and institutional affiliations. An ethical review board may demand a draft or finalized instrument for permission to conduct fieldwork to be given which may not always align to the emergent nature of some designs. For some designs, such as ethnography, extensive fieldwork observations may occur before any commitment is made to formal interview schedules (Roulston & Choi, 2018). In most instances of interview schedule development (and other qualitative instruments), formulation involves the generation of a list of questions and topics that are targeted at addressing the research questions posed, and further speak to the theoretical or conceptual frameworks for the study to be used as an analytical/exploratory lens from which to review findings. The form of a study also plays a role. For instance, there may be nuances in the questions posed for a phenomenological, feminist or ethnographic interview. Moreover, the role of the researcher as a co-constructor of knowledge also needs to be considered in terms of how much their interactions will influence the nature of the flow of the data collection and the areas of focus (Roulston & Choi, 2018). Interview types form a continuum from nonstructured or open-ended, to semi-structured to structured. Non-structured and semi-structured interviews are most prominent in qualitative research. While structured interviews are not readily conducive for qualitative research purposes given their closed-ended nature, non-structured interviews are premised on one open-ended question which is posed to a participant. The participant’s response directs the trajectory of the interview with the researcher as interviewer, then asks further probing questions based on the participant’s response. Nonetheless, the interviewer will still need to have potential follow-up topics in an interview guide (Roulston & Choi, 2018). Semi-structured interviews offer a list of open-ended questions that need to be addressed in an interview. In the use of semi-structured interviews specifically, the topics for discussion are pre-specified and listed in an interview protocol, but the questions can be reworded and do not need to be presented in a set order (Johnson & Turner, 2003). Ideas and issues emerge during the interview that allow the researcher to pursue these leads (Charmaz, 2006). The interviewer will usually formulate these questions based on a target rationale for each question item in terms of answering the overall research questions for a study and in consideration of the conceptual framework for the

study, too. Of course, if the qualitative interviews are a follow-on in terms of sequential mixed methods design in which the qualitative phase succeeds the quantitative phase, then the outcomes of the quantitative phase will steer the development of the instruments for the qualitative phase. Instruments should not be drawn up lightly and the researcher needs to ensure that data resulting from the focal points of these instruments is fit for purpose or congruent with the focus of the research (Barbour, 2018). Generally, researchers will begin with broader questions before moving to specific questions. They will use more openended than closed-ended questions. What the researcher knows about the topic of focus will largely dictate how the conversation unfolds and those researchers with a lot of insider knowledge of the topic will need to reflect on the influence of their preconceptions (Roulston & Choi, 2018). Researchers will utilize interviewing skills such as probing and prompting, reflecting, summarizing and highlighting to elicit detailed, rich descriptions from the participants. Figure 20.2 provides a general overview of considerations in developing an interview schedule. Thereafter, Table 20.2 offers a possible framework that could assist in developing the instruments. Qualitative research as a stand-alone approach is generally inductive in nature with a goal to understand context, participants’ perspectives and generate new perspectives. For qualitative research outputs, data are interpreted by looking for themes grounded in the participants’ responses (Hesse-Biber & Leavy, 2005). There are many ways to analyze qualitative data (see Leech & Onwuegbuzie, 2008 as well as Vanovar et  al., 2021) with thematic analysis and content analysis being prominent. Younas et  al. (2020) stress that the most relevant and robust methods for data analysis should be used. When we look at using the products of the analysis for further instrument development this statement is particularly relevant. Regardless of the analysis method chosen, if we intend to use the analysis findings for development of further instruments then we need to decide on the analysis strategy with this in mind. The methods of data analysis may be inductive or deductive. Deductive methods involve the researcher generating codes from existing literature and adding new or additional codes emerging from the data, making it a hybrid analytic method. A code book is prepared, and codes are developed and refined via the analysis of the data (Mamabolo & Myres, 2019). In this chapter, we focus on inductive grounded theory strategies (taken from grounded theory design) for analysis given their rigorous process which can enhance the credibility of the analysis process. Grounded theory strategies fit with studying the meanings of

HARNESSING MIXED METHODS FOR RESEARCH INSTRUMENT DEVELOPMENT

Outline the broad knowledge areas related to research questions

Develop questions within these broad areas

Think about your respondents and use appropriate language

Take care to phrase questions in a manner that will encourage honesty

Use how questions

Develop probes that will elicit more detail

Begin with warm-up questions

Think about logical flow of the interview and structure accordingly

Ask difficult questions only after rapport has been built

Last question should provide closure

283

Figure 20.2  Interview schedule development considerations Source: Author created.

Table 20.2  Interview schedule framework Interview question Major area Link to research (conceptual links) question What is your experience of professional development activities at your school?

Professional development/ improving practice.

Conceptual framework links

Research question Vocational training 1: What are encourages teachers’ the further experiences development of of mandated staff (Sammons, professional 1999) as development? articulated by in-service training opportunities, updating policies and introducing new programmes (Taggart & Sammons, 1999).

Potential probes and Data analysis follow-up questions strategy What kinds of professional development activities have you completed? How were these activities structured? What benefits were experienced? What were the challenges?

Thematic analysis using grounded theory techniques.

Source: Author created.

participants’ experiences and can be used together with many qualitative approaches such as phenomenology, narrative enquiry, thematic analysis, discursive analysis, participatory action research and, importantly for our purposes, in mixed methods studies. Grounded theory presents a focus on simultaneous data collection and analysis. The iterative process between data collection and analysis allows for the depth of abstraction and precision in emerging themes. Data are analyzed from the start of collection, and, within an inductive logic, inductive theoretical analyses are checked and refined throughout, as well as being grounded in data (Charmaz & Henwood, 2017). Grounded theory comparisons lead to definitions of the characteristics of codes and categories

or themes, help to discern perspectives of the researcher and the participants on the topic of focus, help to make implicit meanings and actions explicit, as well as lead to the drawing of connections between themes and to understandings of the implications of the analyses (Charmaz & Henwood, 2017). The first step in the analysis process is to initiate coding of each generated data set. Qualitative coding is a first analytical step towards moving beyond concrete statements in the data to making analytic interpretations. Coding can be seen as the groundwork to analysis that prepares the way for a much more intensive study (Potter & Wetherell, 1987) by shaping an analytic frame from which to build analysis. Coding “fractures data into

284

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

concepts and categories” (Henning et  al., 2004, p. 131) and entails categorizing segments of data with a short name that simultaneously summarizes and accounts for each piece (Charmaz, 2006). Grounded theory coding particularly involves an initial phase in which each word, line or segment of data is named, and a focused, selective phase in which the most significant or frequent initial codes are used to sort, synthesize, integrate and organize large amounts of data. However, contrary to a quantitative logic that applies preconceived codes to the data, codes are created from what is seen in the data (Charmaz, 2006; Potter & Wetherell, 1987) via inductive logic. As coding has a pragmatic rather than an analytic function, it is recommended that this process of collecting information for analysis should be done as inclusively as possible. After the initial coding is completed, more selective, conceptual and directed coding is undertaken. This focused coding involves using the most significant and/or frequent earlier codes to filter through large amounts of data. Decisions are made as to which initial codes make the most analytical sense to allow for more incisive categorization of the data (Charmaz, 2006). A kind of implicit quantification is present in this process, as a theme is more likely to be identified the more times the phenomenon it signifies is represented while coding (Bryman, 2004). When the collection and coding of additional data no longer leads to new insights for a specific category—a point of data saturation—a summary of each of the categories or themes elicited is described (Pidgeon & Henwood, 2004; Rudestam & Newton, 2007). In 1985, Lincoln and Guba spoke of ensuring the trustworthiness of qualitative research, and later indicated that this trustworthiness involved credibility, dependability, transferability and confirmability (Tobin & Begley, 2004). The credibility of a study is viewed as the fit between participants’ views and the researcher’s representation of these views. Credibility can be demonstrated by means of strategies such as triangulation, member checks and audit trails, among others (Tobin & Begley, 2004). As another quality check, participants should have the opportunity to review, corroborate and revise the research findings, should they deem it necessary, through a process of member validation (Barone, 2004; Bryman, 2004). The aim of member validation is to seek corroboration of the account that is arrived at. In terms of instrument development. This would entail participants having input into the proposed selection of questionnaire items derived from qualitative analysis. Dependability can be ensured by making sure that the research is logical, traceable and clearly documented—in other words, by creating an audit trail. The creation of an audit trail also means that

confirmability or authentication of the interpretation can be achieved (Tobin & Begley, 2004). Anfara et al. (2002, p. 28) reinforce the value of a presentable database by stating that “providing access to the decisions that are made in the process of conducting qualitative research is part of responding to the question of whether or not the findings are sufficiently credible and trustworthy”. Researcher reflexivity also aids dependability, as the researcher keeps a self-critical account of the research process and their influence on the research process (Tobin & Begley, 2004).

MIXED METHODS INSTRUMENT DEVELOPMENT DESIGNS: MAPPING THE LANDSCAPE There have been several developments over the last decade regarding mixed methods design considerations and the manner in which mixing can and should happen. The purpose of mixing may also differ (Greene et al., 1989). For the purposes of our discussion, we focus on a development purpose that seeks to use the results from one method to help develop or inform the other method, and specifically with the focus on instrument construction (Greene et al., 1989). Daigneault and Jacob (2014) highlight that there is a need for the continued discussion of how to use mixed methods research approaches in instrument development and the validation thereof. We would further argue that the instrument development process has implications for legitimation in mixed methods research. It should also perhaps be noted that the way a design is implemented can be decided using a taxonomy model or a typological orientation to design. The designs developed by Creswell and colleagues typically follow a typology orientation (for a detailed description of the evolutions of their thinking about designs, see Chapter 2, this volume). The typology approach has been criticized and several scholars such as Poth (2018) and Creamer (2018), to name a few, have suggested alternatives. However, for the purposes of this discussion, a typology approach is discussed. For instrument development, specifically Creswell and Plano Clark (2018) have mapped typical instrument development designs that have emerged (see Figure 20.3); while these designs are not discussed in depth, several observations can be put forward. The first observation is that instrument development designs are usually sequential. The second observation is that the designs also primarily make use of qualitative data to develop quantitative instruments; this would mean conducting

HARNESSING MIXED METHODS FOR RESEARCH INSTRUMENT DEVELOPMENT

285

Instrument development designs

Greene, Caracelli and Graham (1989)

Steckler, McLeroy, Goodman, Bird, and McCormick (1992)

Creswell (1999)

Creswell, Plano Clark, Gutman and Hanson (2003)

Figure 20.3  Examples of instrument development designs in mixed methods research Source: Author created.

focus groups or interviews before developing a structured questionnaire (Koskey et  al., 2018; Onwuegbuzie et  al., 2010; Steckler et  al., 1992). Of course, another observation is that mixed methods approaches are used to facilitate triangulate or corroborate data obtained using both qualitative and quantitative techniques (Greene et al., 1989). What is far less prominent is the use of quantitative data when developing qualitative instruments or even qualitizing quantitative data. Saldaña (2021) contends that the conversion of statistical results into qualitative forms consolidates empirical materials. This can also be extended to instrument development. An example of this would be to take the Rasch analysis or even factor analysis typically used to explore construct-related validity and transform these into qualitative questions included in focus groups, observation protocols and interviews (although this can be extended to a range of instruments). A more subjective process would be for the researcher to select information from a quantitative analysis, whether via descriptive or inferential statistics, and seek to explore these data further qualitatively for purposes determined by the researcher by including this information in qualitative instrument items. Onwuegbuzie and Leech (2021) state that there is little methodological guidance in qualitizing quantitative data. This means that choices made by researchers in this regard may be undocumented, leading to legitimation concerns for instrument development. It is clear that more guidelines need to be established that draw on innovative analysis techniques that have to be considered during the design phase. There must be alignment between quantitative and qualitative phases for instrument development purposes to make instrument fidelity explicit. This would include thinking through the methodological norms for each approach and alignment with strategies for validity, reliability and trustworthiness. This would mean drawing more clearly on specification frameworks as provided in Table 20.3 for quantitative instruments and Table 20.4 for

qualitative instruments, as well as providing guidelines on the documentation of these processes for audit trail purposes. In both instrument development processes, the process would begin with the conceptualization of the construct or major area in which foci are identified and described. The initial instrument would then be developed and piloted. Changes are then made to the instruments. Inferences regarding the instruments are validated by means of quantitative and qualitative analysis, as well as qualitative dominant cross-over and quantitative dominant cross-over (qualitizing and quantitizing). As an example of this process, Scherman and Liebenberg (2021) demonstrate specifically how the lived experiences of research participants as captured in phenomenology can be used in the quantitative instrument development process. The analysis in descriptive phenomenology results in the development of a list of significant statements that can then be transformed into the items for use in a quantitative instrument. The list of statements is also brought together to form meaningful themes, which then align with what could be understood as constructs from a quantitative perspective. Once the items and constructs have been developed from the qualitative data, an argument is made to test the construct validity of items by means of modern test theory. Scherman and Liebenberg (2021) also quantify qualitative data to undertake statistical analyses for instrument development purposes, which is called quantitizing by Onwuegbuzie and Johnson (2021). The example provided speaks directly to how the legitimation claims can be supported and where full integration is made possible. Central to the legitimation of inferences is the ethical and appropriate use of both qualitative and quantitative instruments, and this starts with the manner in which we design our instruments (Scherman & Liebenberg, 2021), echoing what Onwuegbuzie and Leech (2021) call full integration where 1 + 1 = 1.

286

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 20.3  Specifications for quantitative instruments aligned with qualitative data Construct

Definition

Number of items

Professional Vocational training 26 items development/ encourages improving the further practice. development of staff (Sammons, 1999) as articulated by in-service training opportunities, updating policies and introducing new programmes (Taggart & Sammons, 1999).

Item type

Coding

Possible analysis

Alignment with qualitative data

Dichotomous items Likert scale items.

No = 1 Yes = 2 4 = Strongly agree 3 = Agree 2 = Disagree 1 = Strongly disagree

Frequencies. Cross-tabulations. Scale analysis (including reliability analysis). Correlations.

Word counts. Content analysis.

Source: Author created.

Table 20.4  Specifications for qualitative instruments aligned with quantitative data Major area

Definition

Instrument sources

Possible types of analysis strategy

Alignment with quantitative data

Professional development/ improving practice

Vocational training encourages the further development of staff (Sammons, 1999) as articulated by in-service training opportunities, updating policies and introducing new programmes (Taggart & Sammons, 1999).

Unstructured and semi-structured individual interviews. Focus groups. Document analysis (policy, training plans, training materials). Observations of training opportunities.

Thematic analysis. Discourse analysis. Grounded theory analysis. Narrative analysis.

Inductive construct, concept generation for quantitative item development and analysis via descriptive and inferential statistics where appropriate. Reliability analysis, Rasch/Factor analysis.

Source: Author created

CONCLUSION In this chapter, we have elaborated on how instruments are developed from a single method design and how these can be adapted for mixed methods processes. There have, however, been several debates around how mixed methods designs should be considered. The suggestions in this chapter—namely, using frameworks as an integral part of the development process—are applicable to both partial and full integration.

However, to understand how, when and where to integrate data, you need to understand the design you are using. We have stated that our point of departure is a clear and transparent chain of reasoning. What we mean by this is that the way we design our instruments has a direct bearing on the type of data that is collected, which will have implications for integrating data. While the word “validity” is not often used within qualitative research, the point is that the credibility of our findings also rests upon the development of our

HARNESSING MIXED METHODS FOR RESEARCH INSTRUMENT DEVELOPMENT

instruments. We have not necessarily proposed a design to be used; however, we do highlight the need for more thoughtful development within mixed methods research. We are of the opinion that by including a qualitative component and quantitative approaches to instrument design, it increases not only the instrument fidelity, but also further adds to legitimation claims. Researchers do require competence in qualitative and quantitative analytical techniques. For this reason, researchers should not only understand the development process, but also understand the range of analytical techniques and what can be accomplished when reflecting on the instrument development process. Of course, mixed analysis and cross-over analysis are appropriate in every mixed methods study and will strengthen legitimation claims. We are also suggesting, when considering integration for instrument development processes, that the range be considered such that separate discussions of the different phases identified in the research are described in terms of synthesizing data, combining data or converting data from one form to another. In this chapter, we have provided the instrument blueprints that can be used as part of the mapping process in terms of synthesizing, combining or converting data. These are important planning tools to simplify some of the complexity within mixed methods designs. It should also be noted that methodological norms for both qualitative and quantitative research must be adhered to in the research, thus exploring validity and reliability as described in this chapter and for using member checking as peer debriefing as part of credibility in the process of instrument development and refinement. Several mixed methods legitimation types should also be made explicit, where legitimation is the continuous process reflecting on what happened at each stage of the research process as opposed to just the outcome. The inside–outside legitimation is highlighted in which the researcher presents and appropriately uses the insider’s and observer’s view to deepen understanding, description and explanation. Conversion is another prominent legitimation type, relevant for instrument development, which is associated with qualitizing and/or quantitizing the data, and is conceptualized as the extent to which this process leads to strong meta-inferences. Multiple validities as mixed methods legitimation should also be included and understood as the extent to which addressing legitimation of both quantitative and qualitative components yields high-quality meta-inferences. Whether we analyze numbers or words, we need appropriate processes and tools to make meaning from these data. In this chapter, we have highlighted some tools that can be used when thinking about the development of

287

instruments and how these align within a mixed methods framework. One of the major points of departure is that regardless of the design that we are using, a one-size-fits-all approach will not work. The approach elaborated on in this chapter forces a more explicit approach to the way we plan to develop instruments in terms of how our instruments are aligned to our research question, theoretical framework and, ultimately, mixed analysis.

WHAT TO READ NEXT Scherman, V., & Liebenberg, L. (2021). Item response theory integrating qualitative data. In AJ Onwuegbuzie & R.B Johnson (Eds), The Routledge Reviewer’s Guide to Mixed Methods Analysis (pp. 117–124). Routledge.

In this chapter, the authors address how qualitative data can be quantitized for the purpose of item analysis to enhance validity inferences. The Chapter highlights that there have been no mixed methods designs that pertinently explore how qualitative data can be used to formulate items and item categories. Once qualitative is used for this purpose, item response theory, and Rasch analysis specifically, can be used to explore the psychometric properties of the items before the piloting phase of instrument development. Thus, an additional layer to exploring the validity inferences is added. The suggested reading thus provides the practical application of the guidelines suggested. Koskey, K. L. K., Sondergeld, T. A., Stewart, V. C., & Pugh, K. J. (2018). Applying the mixed methods instrument development and construct validation process: The transformative experience questionnaire. Journal of Mixed Methods Research, 12(1), 95–122. https://doi.org/10.1177/ 1558689816633310

The article included for further reading includes the Instrument Development and Construct Validation (IDCV) model that was first proposed by Onwuegbuzie, Bustamante and Nelson in 2010. While the original model draws on classical test theory, the article included for further reading makes use of modern test theory in a similar manner as the chapter by Scherman and Liebenberg, but also makes use of cognitive interviews. The article provides a practical example of how the IDCV model is applied in the development of the Transformative Experience Questionnaire (TEQ). The authors found that utilizing the IDCV model yielded support for content-, construct- and concurrent-related validity evidence.

288

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Mertens, D. M. (2018b). Mixed methods evaluation designs for instrument development. In Mixed methods design in evaluation (pp. 55–82). https:// dx.doi.org/10.4135/9781506330631

The chapter by Mertens provides an overview of mixed methods and instrument development before providing examples of different ways in which instruments can be developed within a mixed methods framework. Mertens highlights the fact that guidance rarely extends to how to use a mixed methods design for instrument development, as well as how the orientation differs based on the branch of the evaluation theory that evaluators use. To overcome this limitation, Mertens details practical examples and key take-aways for the methods, values, use, social justice and dialectical pluralism branches of evaluation.

REFERENCES Anderson, L. W. (Ed.), Krathwohl, D. R. (Ed.), Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (complete edition). Longman. Anfara, V. A., Brown, K. M., & Mangione, T.L. (2002). Qualitative analysis on stage: Making the research process more public. Educational Researcher, 31(7), 28–38. Barbour, R. (2018). Quality of data collection. In The Sage handbook of qualitative data collection (pp. 217–230). SAGE. https://dx.doi.org/10.4135/ 9781526416070 Barone, D. M. (2004). Case-study research. In Duke, N. K. & Mallette, M. H. (Eds.). Literacy research methodologies (pp. 7–27). The Guilford Press. Bearss, K., Taylor, C.A., Aman, M.G., Whittemore, R., Lecavalier, L., Miller, J., Pritchett, J., Green, B., & Scahill, L. (2016). Using qualitative methods to guide scale development for anxiety in youth with autism spectrum disorder. Autism, 20(6), 663–672. https://doi.org/10.1177%2F1362361315601012 Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Lawrence Erlbaum Associates. Bryman, A. (2004). Social research methods (2nd ed.). Oxford University Press. Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. Sage. Charmaz, K., & Henwood, K. (2017). Grounded theory methods for qualitative psychology. In The SAGE Handbook of qualitative research in psychology (pp. 238–256). SAGE. https://dx.doi.org/ 10.4135/9781526405555

Connell, J., Carlton, J., Grundy, A., Buck, E.T., Keetharuth, A.D., Ricketts, T., Barkham, M., Robotham, Rose, D., & Brazier, J. (2018). The importance of content and face validity in instrument development: Lessons learnt from service users when developing the Recovering Quality of Life measure (ReQoL). Quality of Life Research, 27, 1893–1902. https://doi.org/10.1007/s11136-018-1847-y Creamer, E. G. (2018). An introduction to fully integrated mixed methods research. Sage. Creswell, J. W. (1999). Mixed-method research: Introduction and application. In G. Cizek (Ed.), Handbook of educational policy. Academic Press. Creswell, J. W., & Plano Clark, V.L. (2018). Designing and conducting mixed methods research. Sage. Creswell, J. W., Plano Clark, V. L., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 209–240). Sage. Daigneault, P. & Jacob, S. (2014). Unexpected but most welcome: Mixed methods for the validation and revision of the participatory evaluation measurement instrument. Journal of Mixed Methods Research, 8(1), 6–24. https://doi.org/10.1177%2F1 558689813486190 David, S.L., Hitchcock, J.H., Ragan, B. Brooks, G., & Starkey, C. (2018). Mixing interviews and Rasch modeling: Demonstrating a procedure used to develop an instrument that measures trust. Journal of Mixed Methods Research, 12(1), 75–94. https://doi.org/10.1177%2F1558689815624586 Field, A. (2017). Discovering statistics using IBM SPSS statistics. Sage. Fredricks, J. A., Wang, M., Linn, J. S., Hofkens, T. L., Sung, H., Parr, A. & Allerton, J. (2016). Using qualitative methods to develop a survey measure of math and science engagement. Learning and Instruction, 43, 5–15. https://doi.org/10.1016/ j.learninstruc.2016.01.009 Gay, L. R., Mills, G. E., Airasian, P. W. (2012). Educational research. Pearson. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. Gronlund, N. E. (1998). Assessment of student achievement. Allyn & Bacon. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage. Henning, E., van Rensburg, W. & Smit, B. (2004). Finding your way in qualitative research. Van Schaik. Hesse-Biber, S., & Leavy, P. (2005). The practice of qualitative research. Sage. Johnson, B., & Turner, L. A. (2003). Data collection strategies in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.) (2003). Handbook of

HARNESSING MIXED METHODS FOR RESEARCH INSTRUMENT DEVELOPMENT

mixed methods in social and behavioural research (pp. 297–319). Sage. Kline, P. (2000). A psychometric primer. Free Association Books. Koskey, K. L. K., Sondergeld, T. A., Stewart, V. C., & Pugh, K. J. (2018). Applying the mixed methods instrument development and construct validation process: The transformative experience questionnaire. Journal of Mixed Methods Research, 12(1), 95–122. https://doi.org/10.1177%2F1558689816633310 Krathwohl, D. R. (1998). Methods of educational and social science research: An integrated approach (2nd ed.). Longman. Kvale, S. (1996). Interviews: An introduction to qualitative research interviewing. Sage. Lee, H., Kiang, P., Kim, M., Semino-asaro, S., Colten, M.E., Tang, S. S., Chea, P., Peou, S., & Grigg- Saito, D. C. (2015). Using qualitative methods to develop a contextually tailored instrument: Lessons learned. Asia-Pacific Journal of Oncology Nursing, 2(3), 192–202. DOI: 10.4103/2347-5625.158018 Leech, N. L., & Onwuegbuzie, A. J. (2008). Qualitative data analysis: A compendium of techniques and a framework for selection for school psychology research and beyond. School Psychology Quarterly, 23(4), 587–604. https://doi.org/10.1037/1045-3830. 23.4.587 Lincoln, Y. S. & Guba, E. G. (1985). Naturalistic inquiry. Sage. Mamabolo, A., & Myres, K. (2019). A detailed guide on converting qualitative data into quantitative entrepreneurial skills survey instrument. The Electronic Journal of Business Research Methods, 17(3), 102– 117. https://doi.org/10.34190/JBRM.17.3.001 Mertens, D. (2018a). Mixed methods to address wicked problems. www.ualberta.ca/internationalinstitute-for-qualitative-methodology/webinars/ mixed methods-webinar/archived-webinars (accessed 20 September 2022). Mertens, D. (2018b). Mixed methods evaluation designs for instrument development. In Mixed methods design in evaluation (pp. 55–82). Sage. https://dx.doi.org/10.4135/9781506330631 Messick, S. (1993). Foundations of validity: Meaning and consequences in psychological assessment. Educational Testing Service. Murphy, K. R., & Davidshofer, C. O. (1994). Psychological testing: Principles and applications. PrenticeHall. Nastasi, B. K., & Hitchcock, J. H. (2016). Mixed methods research and culture-specific interventions: Program design and evaluation. Sage. Onwuegbuzie, A. J., Bustamante, R. M., Nelson, J. A. (2010). Mixed research as a tool for developing quantitative research instruments. Journal of Mixed Methods Research, 41(1), 56–78. Onwuegbuzie, A. J., & Johnson, R. B. (2021). Mapping the emerging landscape of mixed analysis. In

289

A. J. Onwuegbuzie and R. B. Johnson (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 1–24). Routledge. Onwuegbuzie, A. J., & Leech, N. L. (2021). Qualitizing data. In A. J. Onwuegbuzie and R. B. Johnson (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 141–150). Routledge. Pidgeon, N., & Henwood, K. (2004). Grounded theory. In M. Hardy & A. Bryman. (Eds.). Handbook of data analysis (pp. 625–648). Sage. Poth, C. N. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Potter, J., & Wetherell, M. (1987). Discourse and social psychology: Beyond attitudes and behaviour. Sage. Roulston, K., & Choi, M. (2018). Qualitative interviews. In The Sage handbook of qualitative data collection (pp. 233–249). Sage. https://dx.doi. org/10.4135/9781526416070 Rudestam, K. E. & Newton, R. R. (2007). Surviving your dissertation (3rd ed.). Sage. Saldaña, J. (2021). Coding techniques for quantitative and mixed data. In A. J. Onwuegbuzie and R. B. Johnson (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 151–160). Routledge. Sammons, P. (1999). School effectiveness: coming of age in the 21st century. Management in Education, 13(5), 10–13. Scherman, V. (2016). Methodological standards and fit for purpose: Criteria to evaluate psychological tests and assessments. In R. Ferreira (Ed.), Psychological assessment: Thinking innovatively in contexts of diversity (pp. 72–85). Juta. Scherman, V., & Liebenberg, L. (2021). Item response theory integrating qualitative data. In A. J. Onwuegbuzie & R. B. Johnson (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 117–124). Routledge. Shannon-Baker, P. (2018). Introducing mixed methods in courses on research design. www.ualberta.ca/ international-institute-for-qualitative-methodology/ webinars/mixed methods-webinar/archived-webinars (accessed 20 September 2022). Steckler., A., McLeroy, K.R., Goodman, R.M., Bird, S.T., & McCormick, L. (1992). Toward integrating qualitative and quantitative methods: An introduction. Health Education Quarterly, 19(1), pp. 1–8. doi: 10.1177/109019819201900101. PMID: 1568869. Suen, H. K. (1990) Principle of test theories. Lawrence Erlbaum Associates. Taggart, B., & Sammons, P. (1999). Evaluating the impact of a raising school standard initiative. In R. J. Bosker, B. P. M. Creemers & S. Stringfield (Eds.), Enhancing educational excellence, equity and efficiency: Evidence from evaluations of systems and

290

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

school change (pp.137–165). Kluwer Academic Publishers. Tran, V. T., Nguyen, T. H., & Chan, K. T. (2017). Developing cross-cultural measurement in social work research and evaluation (2nd ed.). Oxford University Press. Tobin, G.A. & Begley, C.M. (2004). Methodological issues in nursing research. Methodological rigour within a qualitative framework. Journal of Advanced Nursing, 48(8), 388–396.

Vanovar, C., Mihas, P., & Saldaña, J. (2021). Analyzing and interpreting qualitative research after the interview. Sage. Wilson, M. (2005). Constructing measures using and item modeling approach. Taylor & Francis. Younas, A., Rasheed, S. P., Zeb, H., & Inayat, S. (2020). Data integration using the building technique in mixed methods instrument development: Methodological discussion. Journal of Advanced Nursing, 76, 2198–2207. https://doi.org/10.1111/jan.14415

21 Mixed Methods-Grounded Theory: Best Practices for Design and Implementation Michelle C. Howell, Wayne A. Babchuk and Timothy C. Guetterman

INTRODUCTION Mixed Methods-Grounded Theory is a rapidly expanding research design across disciplines and settings that integrates qualitative and quantitative approaches to generate theory. The term “Mixed Methods-Grounded Theory” (MM-GT) was originally coined by Johnson et al. (2010) who not only identified a naturally occurring intersection of mixed methods research (MMR) and grounded theory (GT), but also initiated a conversation among MMR scholars about this intersection that has continued to build momentum among contemporary scholars. However, this literature base has lacked a synthesis of these recent developments useful for researchers to design and implement MM-GT research (see also Chapter 12, this volume). The purpose of this chapter is to extend the theoretical and practical understanding of MM-GT by first providing an overview of MM-GT and its development within the mixed methods literature and then advancing a set of strategies, guidelines, or best practices to help guide MM-GT research. To accomplish these goals, we begin by tracing the development of MM-GT through a review of five key methodological articles in addition to Johnson et  al.’s (2010) work that continued this dialogue highlighting the methodological contributions of

each to the development of MM-GT (Creamer, 2018; Creamer & Edwards, 2019; Guetterman et  al., 2019; Johnson & Walsh, 2019; Walsh, 2015). We then report findings from a systematic methodological review of MM-GT research articles that details the procedures used by their authors in practice and provide recommendations to guide its future applications. This chapter is organized as follows. This chapter begins with a brief introduction to grounded theory methodology leading into a summary of the development of MM-GT through a concise overview of these six key MM-GT articles mentioned above, highlighting the methodological contributions of each to the development of MM-GT. We then conduct a systematic methodological review of empirical MM-GT studies using a MM-GT checklist developed by Guetterman et al. (2019) to identify grounded theory and mixed methods components that are present or absent from these studies that serves to critically examine how MM-GT has been used by these researchers. Building on this work and exemplars that illustrate elements from the MM-GT checklist, we provide recommendations for researchers interested in strengthening their MM-GT research. We conclude by discussing future directions for the continuing development of MM-GT research.

292

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

DEVELOPMENT OF THE MIXED METHODS-GROUNDED THEORY DESIGN Mixed methods-grounded theory is an approach to research that intersects a mixed methods design with a grounded theory design with the intent of developing a theory, model or framework. A theory or model explains a process or action while a framework provides a way of conceptualizing a phenomenon of interest. Its development can be traced through six key publications to date that represent the introduction and refinement of MM-GT for different research applications. We now introduce grounded theory designs broadly before discussing the developments.

Introducing Grounded Theory Designs Grounded theory was first developed and presented in the “death and dying trilogy” (Babchuk & Boswell, 2023, p. 3) by Glaser and Strauss (1965, 1967, 1968). These researchers introduced this methodology with the goal of providing a methodological approach for those interested in developing theory from empirical data collected in the field. In The Discovery of Grounded Theory (1967) they outlined core principles of this methodology including constant comparison, theoretical sampling, theoretical saturation, and memowriting that would serve as the methodological cornerstone of this approach. Later work of both Glaser and Strauss advanced their own iterations of this methodology and continued to refocus and extend it, presenting conflicting interpretations of how to design and implement grounded theory research. Subsequent scholars, some of whom were trained by Glaser and Strauss, offered their own interpretations of this methodology and ultimately a “Family of Methods” (Babchuk, 2011; Bryant, 2017; & Bryant & Charmaz, 2007) emerged, each with its own epistemological leanings and modus operandi. Three of the most popular forms of grounded theory are labelled (1) Classic, Emergent or Traditional, associated with works of Barney Glaser; (2) Systematic, advanced by Anselm Strauss and Juliet Corbin; and (3) Constructivist, originally developed by Kathy Charmaz (see Babchuk & Boswell, 2023; Bryant, 2017; Charmaz, 2014; Creswell & Guetterman; 2019; Hadley, 2017; and Creswell & Poth, 2018 for an in-depth discussion of these approaches). Contemporary scholars continue to adhere to several of the key features of this methodology (mentioned above), but provide divergent strategies for its design and conduct. Most grounded

theorists, however, believe grounded theory is both a product and process of research that should ultimately result in some form of theoretical explanation, model, framework or typology empirically grounded in the data. Although it is often considered the most used contemporary qualitative approach, it has been used in mixed methods research—and to a much lesser extent in mixed and quantitative research— with considerable success (Babchuk & Boswell, 2023).

Grounded Theory in Practice: Is it Inherently a Mixed Method? (Johnson et al., 2010) Although initial conceptualization of grounded theory included both qualitative and quantitative data types and analysis methods, Johnson et  al. (2010) made a substantial contribution through advancing a mixed methods version of grounded theory that arises from integrating the two forms of data. Their work helped to formalize the approach to research. To our knowledge, they are the first to use the term “mixed methods-grounded theory” and the abbreviation “MM-GT”. They identified similar tenets of grounded theory and mixed methods as arguments for their compatibility for intersection. Furthermore, grounded theory and mixed methods research each bring strengths to conduct research that balances both developing and testing theory. In their view, MM-GT is appropriate for complex process-oriented questions, such as what works, how and under what conditions. They presented five potential scientific contributions of MM-GT: (a) linking research to theories; (b) producing practical theories; (c) understanding causation; (d) understanding mediation and moderation including context; and (e) both developing and testing models. Their work conceptualized MM-GT and its potential. What remained was the need for practical guidance for researchers using MM-GT.

Using Quantitative Data in MixedDesign Grounded Theory Studies (Walsh, 2015) Walsh (2015) advocated for a specific type of MM-GT, based on classical grounded theory (i.e., associated with the teachings of Barney Glaser) and named the approach “mixed-design grounded theory”. The approach is similar to mixed methods-grounded theory, but intentionally uses the unique term “mixed design” to uncouple the

MIXED METHODS-GROUNDED THEORY

approach from epistemological considerations and take into account the use of grounded theory in other multimethod designs. Walsh (2015) identified different design approaches based on whether the qualitative and quantitative components were independent and linked or more fully integrated and embedded. In doing so, they advanced specific MM-GT designs rooted in a Glaserian approach to offer practical guidance to researchers.

Enlarging the Conceptualization of Mixed Methods Approaches to Grounded Theory with Intervention Research (Creamer, 2018) Creamer (2018) addressed the specific application of MM-GT to intervention research. Her work advanced yet another intent for applying this design and another novel design type. Specifically, she advanced fully integrated mixed methodsgrounded theory that is distinguished by multiple types of integration throughout the entire research (i.e., not simply when developing inferences). Applying the concept of fully integrated mixed methods research, in fully integrated MM-GT, integration is threaded throughout the research process. Integration occurs through the design, through the analysis process and examining discordance, and through developing a model from both qualitative and quantitative data. Reviewing exemplar articles, Creamer (2018) found key uses of MM-GT to develop a more detailed, multilayered theory, to explore unanticipated intervention outcomes, and to develop fully integrated grounded theory models that include quantitively and qualitatively derived paths and mediators. The final exemplar reinforced the use of MM-GT for both theory development and testing.

Contemporary Approaches to Mixed Methods-Grounded Theory Research (Guetterman et al., 2019) Guetterman et  al. (2019) systematically explored the use of Mixed Methods-Grounded Theory (MMGT) across disciplines to see how contemporary scholars were using this approach with the overall goal of developing recommendations for successfully conducting MM-GT. These researchers conducted a search of Academic Search Premier and its available databases, and identified 61 articles that met their inclusion criteria (i.e., use of quantitative and qualitative empirical data, and grounded

293

theory employed in the qualitative strand). Using a code book they designed for this task, focusing on general information, mixed methods, and grounded theory features, they coded these articles to assess researchers’ use of MM-GT. Three major findings emerged from this analysis across their dataset, including that (1) often little methodological detail was provided; (2) there was a general lack of actual theoretical development even though a requisite feature of grounded theory is to generate some form of a theoretical explanation; and (3) that the vast majority of articles they reviewed (73 per cent) employed a convergent mixed methods design as opposed to explanatory sequential and exploratory sequential designs or their variants. They concluded that there is a general lack of consistency among researchers in how they design, implement and integrate MM-GT in the literature and none of the published work represented the full potential of MM-GT as outlined by Johnson et  al. (2010) in their seminal work on MM-GT. Based on the published literature and their own work on MM-GT, Guetterman et  al. (2019) provided initial recommendations to better use of this approach that can avoid or sidestep some of the pitfalls noted in the contemporary studies they reviewed. What remained was the need to refine best practices, consider their practicality and to account for the numerous creative uses of MM-GT.

Embedding Dialogic in Mixed Methods Approaches to Theory Development (Creamer & Edwards, 2019) Creamer & Edwards (2019) explored the role of dissonance in MM-GT and thus advanced both an intent for selecting the design and a procedure for theory development. Rather than viewing it as unfavourable, their approach embraced dissonance between findings from different sources of data in mixed methods research. Focusing on dialogical forms of mixing, they described analytical procedures geared at making sense of this dissonance that can contribute to the development or revision of theory. Drawing upon Walsh’s (2014; and see Johnson & Walsh, 2019) insightful discussion of “incremental” theorizing that refers to the process of building incrementally on theory employing existing concepts or constructs, and “rupture” theorizing that refers to theorizing that provides a new perspective on a studied phenomenon or theory borrowed from different domains, these researchers identified seven empirical mixed methods articles that specifically focused on understanding dissonance between the quantitative and qualitative data strands related to the development of theory.

294

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Looking at five features for each of the “critical cases” (p. 243), they examined five of the seven articles which positioned their research results in a manner that they found to be consistent with rupture theorizing. According to Creamer and Edwards (2019), another unexpected finding of this research was the creative and innovative ways the authors of the critical case research articles employed dialogic mixing to reach a new understanding of the juxtaposition of qualitative and quantitative data findings. These understandings, they argued, challenged conventional thinking about a core construct and the way it has been traditionally viewed. A central thesis of this work is that dialogical mixing and rupture theorizing have the potential to effectively contribute to theory building across settings in mixed methods research.

Mixed Grounded Theory (Johnson & Walsh, 2019) Johnson and Walsh (2019) collaborated to advance and introduce a more inclusive vision for MM-GT. Although many scholars and practitioners often consider grounded theory to be a primarily qualitative methodology as mentioned earlier, Johnson & Walsh (2019) underscored that since its early development by Glaser and Strauss (1967) in The Discovery of Grounded Theory, this design can and has been used in qualitative, quantitative, mixed methods and multimethod research. Casting a wider net than its application in Mixed MethodsGrounded Theory (MM-GT,) Johnson and Walsh (2019) advanced the term Mixed Grounded Theory (MGT) to encompass the use of grounded theory in mixed methods research in which researchers integrate qualitative and quantitative research approaches, concepts, philosophies, etc., and in more broadly based multimethod research that is a term for research that uses multiple methods or methodologies in a complementary fashion to better understand the studied phenomenon. These authors therefore viewed “mixed research” or MGT as a more inclusive term that encompasses a more expansive merger of grounded theory and the mixed methods and multimethod movements that “suggests the mixing or interplay of differing perspectives, methods, methodologies, paradigms, and so forth” (p. 52). They argued that traditionally conceived mixed methods research (MMR) fails to account for the mixing of multiple methods, methodologies and paradigms within qualitative or quantitative research. MGT draws upon the use of grounded theory and mixed research traditions and is both an approach to research and an outcome (i.e., MGT), as is, for that matter, qualitative

grounded theory or mixed methods-grounded theory. These authors outlined five ways to produce an MGT and provide an illustration of several MGT research designs (i.e., three core mixed methods research designs further subdivided into six MGT designs) in the literature. As was the case with other MM-GT articles reviewed in this chapter, Johnson and Walsh (2019) encouraged and lauded the potential of using grounded theory in new and creative ways in mixed methods and multimethod research.

Synthesis: Where We have Come and Where to Go Next These six major publications have advanced the concept of MM-GT, identified new intents for MM-GT, and discussed specific types of MM-GT designs and procedures. These methodological articles were largely theoretical in advancing the design and illustrating potential uses. Continuing our previous work, we sought to understand how studies explicitly framed as MM-GT are being conducted.

CHARACTERISTICS OF MM-GT Procedures Involved in Identifying the Key Characteristics Given the growing interest in MM-GT, we conducted a systematic methodological review, following procedures recommended by Howell Smith and Shanahan Bazis (2021), of empirical MM-GT studies that explicitly positioned themselves as mixed methods-grounded theory. The aim of conducting the review was to identify key characteristics of MM-GT and develop best practices to extend our previous recommendations. We operationalized this positioning as any empirical study that cited at least one of the six methodological articles we summarized above, where the citation was directly related to MM-GT. In other words, studies that cited a key MM-GT article, but the citation was regarding a core mixed method design or other topic, were not included in our review. We used the “Cited By” feature in Google Scholar and identified 435 potential records. After removing duplicates, we screened all titles and abstracts for evidence that the record was a journal article and an empirical study. We then downloaded the complete text for the remaining 42 articles for evidence that the study was intended by the authors to be a mixed methods-grounded

295

MIXED METHODS-GROUNDED THEORY

theory study. After the full text review, 16 remaining MM-GT empirical studies were included in our methodological review. It is important to note that our pool of 16 articles does not represent all studies conceptualized as MM-GT (e.g., Walsh, 2014). Because we were interested in exploring how MM-GT studies that cited the key MM-GT articles addressed MM-GT features, we coded each study using the same framework established in Guetterman et al. (2019).

specified in Table 21.2. These characteristics include citing grounded theory references, identifying the grounded theory approach, matching research questions to the approach, describing sampling, saturation, memoing (or memo-writing), grounded theory analysis methods, generating a theory, and attending to validity and quality. Overall, we noticed trending improvement, comparing these characteristics in this review as compared to our 2019 examination of MM-GT studies.

General Characteristics of MM-GT Studies

Mixed Methods Research Characteristics of MM-GT Studies

The 16 MM-GT studies in our review represented a variety of disciplines, with the majority of the topics related to education and health. The journals publishing the MM-GT studies were all unique, with the exception of Sustainability and the Journal of Mixed Methods Research, which each published two MM-GT studies. Authors of the studies in our pool represent institutions from around the world. Only seven of the studies were authored by researchers affiliated solely in the United States. Although each of the six key MM-GT articles generated records when using the cited by feature in Google Scholar, not all of them were cited by our pool of empirical MM-GT studies. Table 21.1 shows how many times each of the key MM-GT articles were cited in our pool. Individually, the empirical MM-GT studies cited from one to five of the key MM-GT articles, with a mean of two citations per study. The citations might merely reflect the journals in which the articles were published.

As a best practice for MM-GT studies, we also recommend identifying mixed methods specific features in our previous review (Guetterman, 2019). These features include citing appropriate mixed methods literature, explaining the rationale for using mixed methods, specifying the mixed methods design that is consistent with the research questions, achieving integration, including a procedural diagram, and discussing value-added, legitimation, and quality indicators (Table 21.3). Consistent with our previous findings, we found that most mixed methods designs in MM-GT studies reviewed were convergent designs, which raises the question that other designs may be under-utilized. For example, an exploratory sequential mixed methods design that begins by qualitatively developing a theory followed by quantitatively testing the theory could be an ideal fit for many MM-GT studies. Yet, exploratory sequential designs have consistently been the least common mixed methods design in our reviews.

Grounded Theory Characteristics of MM-GT Studies

Employing Key Features of MM-GT in Published Studies

Based on our previous review (Guetterman, 2019), we recommend as a best practice that MM-GT studies describe its grounded theory characteristics as

Our prior review (Guetterman, et  al., 2019) also advanced key features of MM-GT to include as a

Table 21.1  Key MM-GT article citations in an MM-GT review pool Key MM-GT article Creamer & Edwards, 2019 Creamer, 2018 Guetterman et al., 2019 Johnson & Walsh, 2019 Johnson et al., 2010 Walsh, 2015 Source: Author created.

# of citations in MM-GT studies 0 2 14 5 5 4

% of citations in MM-GT studies 0 13 88 31 31 25

296

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 21.2  Best practices for grounded theory features to include in published MM-GT studies Feature

Description

Grounded theory references

Citing appropriate grounded theory references that are consistent with the type of grounded theory used and methods. Identifying a grounded theory design and specifying the type (e.g., classicalGlaserian, systematic-Corbin and Straus, constructivist-Charmaz). Research questions are consistent with the approach and could reasonably be addressed. Describing the use of theoretical, purposive or another sampling approach to contribute to theory development.

Grounded theory approach identified Research questions match approach Theoretical sampling Saturation Memoing/memo-writing Analysis approach Generated a theory Validation strategies Quality indicators

Discussing the concept of saturation to guide sampling and when to cease data collection. Mentioning memoing, which is a process of documenting analytical ideas during analysis and theory development. Discussing a grounded theory approach to analysis, such as the constant comparative method or grounded theory coding. Presenting a theory, model or framework. Discussing strategies for checking the accuracy of findings and interpretations. Mentioning quality indicators or a specific framework used (e.g., COREQ).

Source: Author created.

Table 21.3  Best practices for mixed methods features to include in published MM-GT studies Feature

Description

MMR references

Citing appropriate mixed methods literature consistent with the design integration strategies and other procedures used. Explaining a rationale for using mixed methods research. Discuss what mixed methods design is being used. Research questions or aims are consistent with the mixed methods design. Discussing strategies or methods of integration and presenting integrated results. Joint displays are one strategy for integration. Drawing a procedural diagram to indicate the flow of procedures. Reflecting on the value-added of using a mixed methods design relative to a monomethod design. Identifying validity threats based on the mixed methods design or ways to ensure legitimation. Describing any quality criteria (e.g., GRAMMS) that have been applied.

Rationale provided MMR design specified Research questions match approach Integration, including joint displays Procedural diagram Mixed methods value added Validation or legitimation strategies Quality indicators Source: Author created.

best practice in published MM-GT studies. We present our findings from our methodological review in addition to illustrations of these features within the pool of articles.

Including methodological references

There is a strong correlation with the number of methodological references and the presence of items from our MM-GT best practices checklist. We ran correlations between the number of GT,

MMR and MM-GT citations with the number of checklist items. As shown in Table 21.4, correlations ranged from 0.69 to 0.80, indicating a strong relationship between these factors. Authors who have more methodological citations also tend to address more items on the MM-GT checklist. For example, in Kretschmer et  al. (2021), they included seven GT citations, two MMR citations, four MM-GT citations and addressed 13 out of 23 checklist items in their MM-GT study of personal and paradigm level factors that influence organic food systems.

297

MIXED METHODS-GROUNDED THEORY

Matching research questions to the approach

Only half of the MM-GT studies in our pool (n = 8) provided a qualitative research question, with only about half of those research questions (n = 5) being aligned with a grounded theory approach (i.e., related to understanding a process, to develop a theory, framework, model, or typology). Given the intent of grounded theory, we would expect research questions that are consistent with these intents. Seven of the eight MM-GT studies that had qualitative research questions also had mixed methods research questions; one study without a specific qualitative research question had a mixed methods research question, bringing the total number of studies with MMR research questions to eight. Unlike the GT research questions, the MMR research questions were all aligned with a mixed methods approach. Overall, only three MM-GT studies had both GT and MMR research questions that aligned with each approach. Krueger et al. (2020) provides an excellent example of research questions that match the methodological approach in their sequential explanatory MM-GT study, exploring how reflective practice is experienced by occupational therapists. The initial quantitative strand of their study used a survey to measure self-reflection behaviour, evidence-based practice (EBP) skill and knowledge, and organizational and practice-based factors associated with EBP implementation. The research questions for their follow-up qualitative strand included the following: “(1) What practice experiences trigger self-reflection behavior in occupational therapists? (2) How do occupational therapists utilize reflection in relation to depth on the critical reflection inquiry model? And (3) How does reflection in participants relate to actions including use of EBP?” (p. 323). The focus on “trigger” experiences and relationship with the critical reflection inquiry model demonstrates alignment with a grounded theory approach. Their overarching MMR research question, “How [may] reflection act as a support to EBP implementation?” (p. 323) calls for the direct integration of

the qualitative (act of reflection) and quantitative (EBP implementation) strands.

Providing rationales for MM-GT

The MM-GT studies in our pool had varying degrees of providing an explicit rationale for their methodological approach with only 56 per cent providing a rationale for using grounded theory and 81 per cent for mixed methods research; only 44 per cent provided a rationale for both approaches. Tsortanidou et  al. (2020) demonstrates clear rationales for both their GT and MMR approaches in their convergent MM-GT study of whether Waldorf-inspired, imaginative teaching methods in partnership with low-technology prototyping can promote effectively new medial literacies skill development among upper elementary school students. They succinctly describe their rationales for the use of MM-GT as our resulting approach uses mixed methods to understand the pedagogical mechanism by capitalizing on the strength of both qualitative and quantitative methods. Additionally, the research questions call for the development of a pedagogical approach (grounded theory) by exploiting qualitative and quantitative data achieving in this way triangulation (mixed methods). (pp. 3–4)

Another example of providing an explicit MM-GT rationale can be found in Garnett et al. (2022) in their convergent mixed methods case study of the experiences and perceptions of elementary school staff and students participating in restorative practices (RP) community building circles. Garnett et  al. (2022) noted their choice to use MM-GT approach for their data analysis was because they sought to “elevate the perceptions, experiences, and feelings of individuals” (p. 5).

Specifying GT approach and MMR design

The pattern of specifying the GT approaches and MMR design almost directly duplicates the specification of rationales. Only 50 per cent of the

Table 21.4  Correlations between methodological citations and mixed methods-grounded theory best practices

# GT citations # MMR citations # MM-GT citations # Checklist items Source: Author created.

# GT citations

# MMR citations

# MM-GT citations

# Checklist items

1.00 0.74 0.77 0.80

1.00 0.79 0.69

1.00 0.78

1.00

298

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

MM-GT studies in our review pool noted a particular approach to GT; of those, three (19 per cent) were Classical (Glaser), three (19 per cent) were Constructivist (Charmaz) and two (13 per cent) were Systematic (Strauss and Corbin). There were 81 per cent of the studies in our MM-GT review pool that identified a specific MMR design as described above. Combined, there were seven (44 per cent) studies that identified both a GT approach and an MMR design. For example, Wiess & Bensimon (2020) used a constructivist GT approach (Charmaz 2000, 2014) in their “concurrent nested” MMR design (Creswell et  al., 2003; Hanson et  al., 2005) that broadened the clinical and theoretical understanding of the use of group music therapy among people uprooted from the Gush Katif settlements in the Gaza Strip. They selected a constructivist approach because it “provided a conceptual model that emerged from the data and was ‘grounded’ in the participants’ words and experiences” (p. 177) and intersected it with a concurrent nested MMR design in order to “confirm, cross-validate, or corroborate” (p. 177) their findings within a single study. Wood (2020), on the other hand, used a systematic grounded theory approach (Corbin & Strauss, 1990) in their “convergent parallel” (Creswell & Plano Clark, 2007) study that examined the social media logics underpinning New South Wales Police Force’s “meme strategy” that employed humour and cute content to increase social media engagement. They selected a systematic GT approach because of its “process of primary, secondary, and selective coding, in which broad initial codes were gradually honed through the constant comparison of data” (p. 45). The convergent parallel MMR design, then, provided an opportunity to examine the qualitative and quantitative data separately before being integrated.

Using GT and MMR procedures

The use of specific GT and MMR procedures differed across our pool of MM-GT studies. Although the precise type of GT coding varied with the GT approach, the use of some form of GT coding was found in a clear majority of MM-GT studies, followed by the frequent use of constant comparison throughout the coding and analytic process. Only about half of the studies described using memoing as part of their procedures. Surprisingly, theoretical sampling and saturation were each only used in a fourth of the studies. A theory, framework, model or typology was only developed in about half of the studies in our pool. Because we consider theory development to be an essential component of any MM-GT study, we present a brief summary in Table 21.5 of the nine theories,

frameworks, models, or typologies developed in the pool of MM-GT studies we reviewed. In terms of MMR procedures, slightly more than half of the studies described multiple points of integration. The study by Shim et  al. (2021), an exploratory-confirmatory MM-GT design, developed a theory of dance/movement therapy for chronic pain management, that was unique in that it featured three different kinds of integration. In their first phase, a meta-model was developed by integrating a formative model based on literature with a grounded theory model based on interviews. The second, confirmatory phase then tested the model via a mixed methods experiment that integrated standardized tests with participant journals and interviews to create yet another model. In the final phase, the meta-model from the first phase was integrated with the model from the second phase to produce the final theoretical model. Of the studies in our MM-GT pool, 44 per cent provided a procedural diagram, which help readers visualize the study design and methods. In their exploratory MM-GT study of differences in the biopsychosocial profiles of patients with cardiac disease in e-usage groups, Anttila (2021) included a procedural diagram that visualizes previous qualitative research informing the first (quantitative) step of their study. The quantitative step is visually nested in the second step, which is a comparative analysis, which, in turn, is visually nested within the third mixed methods step where the biopsychosocial profiles developed by the previous qualitative research are finalized. Gibbons et  al. (2019) also included a procedural diagram of their convergent MM-GT study on the relationships between post-treatment cancer patients and family caregivers. Their diagram used two boxes, one on top of the other, for the quantitative survey strand and the qualitative grounded theory interview strand. They indicated independent analysis for each strand within large arrows that point to a circle that represents the merging of the two strands. An arrow labelled “interpretation” then points towards a final box indicating the mixed methods results. Joint displays that present the integration of quantitative and qualitative elements were only used in 44 per cent of the studies in our MM-GT pool. As illustrated by Puffer & Pence (2020) in their explanatory study of the role of negative emotions in the career decision-making process, they presented a matrix (p. 10) that listed the top six negative emotions evoked by specific career options, along with the frequency in the quantitative data in one column, and then provided a description of that emotion and quotes from participants that describe how that emotion related to their career opportunities, thus illustrating each emotion from the quantitative strand with

MIXED METHODS-GROUNDED THEORY

299

Table 21.5  Theories, frameworks, models or typologies developed in MM-GT studies MM-GT-Study

Theory, framework, model or typology developed

Anttila, 2021 Gibbons et al., 2019

Profiles for individuals changing their lifestyle as part of the rehabilitation process. Emerging model for building dyadic resilience for cancer patients and their family caregivers. Tested model of the process of developing interest in an Engineering PhD. Emerging theory for global organic mindset pattern. Model of the tangible and intangible components of territorial governance associated with coffee production. Theoretical framework that explains the adaptive purposes for negative emotionality within the career decision-making process. Theoretical model of dance/movement therapy for resilience building in people with chronic pain. Theory of the factors that the young people who have served the judicial sentence of living in an open regime educational group in Catalonia for crimes of child-to-parent violence that enhance or limit empowerment. Theory of the process of IT acculturation.

Howell Smith et al., 2020 Kretschmer et al., 2021 Palma et al., 2020 Puffer & Pence, 2020 Shim et al., 2021 Trull Oliva & Soler Masó, 2021

Walsh, 2020 Source: Author created.

qualitative excerpts. Sena Moore & HansonAbromeit’s (2018) convergent MM-GT study of a music therapy intervention for pre-schooler emotional regulation development provides another example with a matrix (p. 432) that began with a column for the core categories derived from the interviews and then illustrated each category with a relevant quote from their qualitative data. The matrix also included correlations and effect sizes from the quantitative data related to each core category, and then notes if the quantitative results were congruent or discrepant from the qualitative findings, as well as providing an explanation for discrepancies. Another example of a joint display is the Machleid et al. (2020) convergent MM-GT study of medical students’ knowledge and opinions of digital health. Machleid et al. quantitized their qualitative findings from a thematic analysis of medical student’s definitions of eHealth and reflect the frequency counts in the proportion of space on their concentric circle display of the interrelatedness of the themes (p. 6). Explicit discussions of the value added by conducting a MM-GT study only appeared in 44 percent of the studies in our pool. For example, in her explanatory MM-GT study of the concept of IT acculturation, Walsh (2020) noted: The qualitative data helped us bring into our quantitative nomological framework the concept of effective use (Burton-Jones and Grange, 2012) and understand the processual quality of the phenomenon we were studying, together with the reciprocal relationships and recursive causal loops between several constructs. (p. 44)

Addressing validity/legitimation and quality indictors

While in principle, addressing issues of validity and quality are critically important in any research study, in practice, they are infrequently addressed in the MM-GT studies in our pool. Addressing issues of validity in terms of qualitative grounded theory was fairly common. However, none of the studies in our pool specifically address indicators of quality. Only one study, Howell Smith et  al. (2020), addressed legitimation and quality indicators for mixed methods. In their sequential exploratory MM-GT study of the process for developing interest in the Engineering PhD, Howell Smith et al. presented a table (p. 189) that detailed how they addressed four types of legitimation: sample integration, inside outside, weakness minimization and multiple validities. Although not presented directly in their publication, they note that the dissertation that their article is based on addressed all five guidelines for Good Reporting of a Mixed Methods Study (GRAMMS) (O’Cathain et  al., 2008), which is one of several quality indicator frameworks for MMR.

Guiding Practices for Mixed MethodsGrounded Theory Design and Implementation Since the publication of Johnson et  al.’s (2010) MM-GT article, the attention on MMR designs has gradually been expanding, as evidenced by the methodological articles that define, synthesize

300

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

and expand their use, which we review here. In this chapter, we provided a summary of six key MM-GT methodological articles and then illustrate the use of elements in the MM-GT checklist (Guetterman et al., 2019) in a pool of 16 MM-GT studies that explicitly position themselves as MM-GT studies. Given the increase in citations of the key MM-GT articles and the inclusion of MM-GT features, we recommend including these features in published MM-GT articles as a best practice. In this chapter, we have attempted to synthesize the development of MM-GT, advance best practices for key features of MM-GT designs, and provide illustrations of these features. We strongly encourage other scholars to help further refine MM-GT. Along these lines, Creamer’s (2021) Advancing Grounded Theory with Mixed Methods is an important and insightful work that should be coupled with other key readings we have reviewed in this chapter to gain a more complete understanding of how to use this approach. Creamer provides practical guidelines and exemplars to help guide potential MM-GT researchers in the social and behavioural sciences. As discussed in this methodological review, researchers would benefit from more systematic adherence to both grounded theory and mixed methods procedures, and need to make these explicit in their published work. We have shown in this methodological review that several of the hallmark grounded theory elements we identified were not employed or had limited use in the MM-GT studies in our pool, such as theoretical sampling, theoretical saturation, memoing, etc., all of which are generally considered key components or essential in all versions or sub-approaches of grounded theory. This may be due in part to challenges to adapting GT to an MMR design but, perhaps more importantly, researchers need to be more systematic and informed regarding how they conduct their research. In this vein, GT validation strategies of some form were usually included, but MMR quality criteria rarely were mentioned. Several authors have explicitly addressed these quality indicators, such as that explicated in the legitimation typology advanced by Onwuegbuzie and Johnson (2006; see also Johnson and Christensen 2017, 2020; Onwuegbuzie & Johnson, 2021; and Perez et al., 2022) to help guide their research. In our earlier publication (Guetterman et  al., 2019), we provided recommendations for conducting MM-GT that we will reiterate and expand upon here. Our more recent methodological review for the purpose of this chapter further underscored our earlier emphasis for the need of more systematic, rigorous, transparent and

detailed use of GT, MMR and MM-GT research. As should appear obvious at this point, we strongly recommend that researchers become more familiar with both the GT and MMR literature through training, mentoring and perseverance, and should become versed in the guiding MM-GT research we review in this chapter. One needs to gain an understanding of the specifics of these approaches and strive for methodological and analytical congruence in formulating research questions, and all other aspects of design and implementation. Researchers should clearly articulate their rationale for selecting MM-GT and how they are going to systematically integrate the qualitative and quantitative strands of their research. Researchers would also benefit from a clear understanding of the different forms of GT and MMR and not only justify their decision-making, but follow suggestions offered by scholars of these approaches to help guide their research. For example, researchers need to adhere to the central components of GT research (e.g., constant comparison, theoretical sampling and saturation, memoing, development of theory) when using this methodology and have a solid grasp of MMR designs as outlined by scholars in this tradition. As we mentioned above, it is important to employ not only strategies for validating the grounded theory findings, but also legitimation strategies to address the potential threat to validity in MMR. We strongly recommend detail and transparency in published reports regarding this methodological and procedural decision-making to help other researchers not only understand the process used to conduct the research, but also to aid them in their own research efforts.

Limitations and Future Directions Although the use of MM-GT is growing, there is a paucity of examples of the intentional intersection between mixed methods and grounded theory. The key methodological articles regarding this approach are important to our understanding of the potential of MM-GT and its application across disciplines and settings. However, the dearth of exemplar MM-GT studies that employ most or all of the GT, MMR and MM-GT components we stress in this review means that there are few models that aspiring mixed methods-grounded theorists can use to direct their research. Also problematic is the relatively frequent lack of detail regarding the methodological decision-making and procedures often provided by researchers that further confound one’s ability to assess the quality of their publications. We are left to wonder that if

MIXED METHODS-GROUNDED THEORY

scant methodological information is the result of less rigorous and informed research designs, or if it is intentionally left out due to journal word limits, or perhaps both. In the future, we expect to see the continued evolution of MM-GT in both methodological and empirical literature. A natural extension of the research designs identified by Johnson and Walsh (2019) would be the development of specific MM-GT designs that align with the researcher’s intent, such as the exploratory–confirmatory MM-GT design used by Shim et  al. (2021). Likewise, we anticipate the development of specific procedures that facilitate the fully integrated mixed method approach to grounded theory described by Creamer (2018). Further, we encourage increased attention on attending to issues of legitimation in MM-GT studies. As these innovations develop, we predict that new ways of intersecting mixed methods and grounded theory will emerge, such as the intersection of case study with MM-GT designs in Garnett et al. (2022) or the intersection of grounded theory ethnography with mixed methods (Babchuk & Boswell, 2023; Babchuk & Hitchcock, 2013) as a natural progression of MM-GT. In this chapter, we have reviewed several foundational MM-GT publications and how their authors have advocated for the potential and use of this methodology, and we have assessed the use of MM-GT in the literature. We offer recommendations for conducting MM-GT that help guide those interested in applying this approach. We hope future work builds and extends on the MM-GT tradition we have discussed in this chapter and that it is embraced by future researchers who continue to strive to conduct more systematic and rigorous MM-GT in their studies.

WHAT TO READ NEXT Johnson, R. B., McGowan, M. W., & Turner, L. A. (2010). Grounded theory in practice: Is it inherently a mixed method?. Research in the Schools, 17(2).

We consider this the foundational article setting the stage for MM-GT research as the authors coin the term “Mixed Methods-Grounded Theory” and discuss its potential use and application. Johnson, R. B., & Walsh, I. (2019). Mixed grounded theory: Merging grounded theory with mixed methods and multimethod research. In A. Bryant, & K. Charmaz (Eds.), The SAGE handbook of current developments in grounded theory (pp. 517–531). Sage.

301

As we outline above, this chapter considers the use of grounded theory in mixed methods and multimethod research. These authors outline ways to produce MGT research and its use in six basic MGT designs. Creamer, E. G. (2021). Advancing grounded theory with mixed methods. Routledge.

Creamer provides a thorough treatment of MMGTM and its history and use. It is geared towards students of MM-GT to provide practical suggestions as to how to conduct this methodology.

REFERENCES Anttila, M. R., Soderlund, A., Paajanen, T., Kivistö, H., Kokko, K., & Sjögren, T. (2021). Biopsychosocial profiles of patients with cardiac disease in remote rehabilitation processes: Mixed methods grounded theory approach. JMIR Rehabilitation and Assistive Technologies, 8(4), e16864. https://doi.org/10.2196/ 16864 Babchuk, W. A. (2023). Grounded theory as a “family of methods”: A genealogical analysis to guide research. US–China Education Review, 1(3): 383–388. Babchuk, W. A., & Boswell, E. (2023). Qualitative research approaches and designs: Grounded theory. In R. Tierney, F. Rizvi, K. Ercikan, & G. Smith (Eds), International encyclopedia of education, Volume 4 (pp. 107–122). Elsevier. https://doi. org/10.1016/B978-0-12-818630-5.11013-9 Babchuk, W. A., & Hitchcock, R. K. (2013). Grounded theory ethnography: Merging methodologies for advancing naturalistic inquiry. In E. P. Isaac-Savage (Ed.), Proceedings of the 54th Annual Adult Education Research Conference (pp. 27–34). Lindenwood University. Bryant, A. (2017). Grounded theory and grounded theorizing. Pragmatism in research practice. Oxford University Press. Bryant, A., & Charmaz, K. (2007). Grounded theory research: Methods and practices. In A. Bryant & K. Charmaz (Eds.), The Sage handbook of grounded theory (pp. 1–28). Sage. Charmaz, K. (2000). Grounded theory: Objectivist and constructivist methods. In N. Denzin & Y. Lincoln (Eds.), Handbook of Qualitative Research (2nd ed., pp. 509–535). Sage. Charmaz, K. (2014). Constructing grounded theory. Sage. Corbin, J. M., & Strauss, A. (1990). Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative Sociology, 13(1), 3–21.

302

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Creamer, E., & Edwards, C. (2019). Embedding the dialogic in mixed method approaches to theory development. International Journal of Research & Method in Education, 42(3), 239–251. https://doi. org/10.1080/1743727X.2019.1598357 Creamer, E. G. (2018). Enlarging the conceptualization of mixed method approaches to grounded theory with intervention research. American Behavioral Scientist, 62(7), 919–934. https://doi. org/10.1177/0002764218772642 Creamer, E. G. (2021). Advancing grounded theory with mixed methods. Routledge. Creswell, J. W., & Guetterman, T.C. (2019). Educational research (6th ed.). Pearson. Creswell, J. W., & Poth, C.N. (2018). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). Sage. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Sage. Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209–240). Sage. Garnett, B. R., Kervick, C. T., Moore, M., Ballysingh, T. A., & Smith, L. C. (2022). School staff and youth perspectives of tier 1 restorative practices classroom circles. School Psychology Review, 51(1), 112–126. https://doi.org/10.1080/2372966X.2020.1795557 Gibbons, S. W., Ross, A., Wehrlen, L., Klagholz, S., & Bevans, M. (2019). Enhancing the cancer caregiving experience: Building resilience through role adjustment and mutuality. European Journal of Oncology Nursing, 43, 101663. https://doi.org/ 10.1016/j.ejon.2019.09.004 Glaser, B. G., & Strauss, A. L. (1965). Awareness of dying. Aldine. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine. Glaser, B. G., Strauss, A. L. (1968). Time for dying. Aldine. Guetterman, T. C., Babchuk, W. A., Howell Smith, M. C., & Stevens, J. (2019). Contemporary approaches to mixed methods–grounded theory research: A field-based analysis. Journal of Mixed Methods Research, 13(2), 179–195. https://doi.org/10.1177/ 1558689817710877 Hadley, G. (2017). Grounded theory in applied linguistics research: A practical guide. Routledge. Hanson, W. E., Creswell, J. W., Plano Clark, V. L., Petska, K. S., & Creswell, J. D. (2005). Mixed methods research designs in counseling psychology. Journal of Counseling Psychology, 52(2), 224–235. https://psycnet.apa.org/doi/10.1037/ 0022-0167.52.2.224 Howell Smith, M. C., & Shanahan Bazis, P. (2021). Conducting mixed methods research systematic

methodological reviews: A review of practice and recommendations. Journal of Mixed Methods Research, 15(4) 546–566. https://doi.org/10.1177/ 1558689820967626 Howell Smith, M. C., Babchuk, W. A., Stevens, J., Garrett, A. L., Wang, S. C., & Guetterman, T. C. (2020). Modeling the use of mixed methods– grounded theory: Developing scales for a new measurement model. Journal of Mixed Methods Research, 14(2), 184–206. https://doi.org/ 10.1177/ 1558689819872599 Johnson, R. B., & Christensen, L. B. (2017). Educational research: Quantitative, qualitative, and mixed approaches (6th ed.). Sage. Johnson, R. B., & Christensen, L. B. (2020). Educational research: Quantitative, qualitative, and mixed approaches (7th ed.). Sage. Johnson, R. B., & Walsh, I. (2019). Mixed grounded theory: Merging grounded theory with mixed methods and multimethod research. In A. Bryant, & K. Charmaz (Eds.), The SAGE handbook of current developments in grounded theory (pp. 517–531). Sage. Johnson, R. B., McGowan, M. W., & Turner, L. A. (2010). Grounded theory in practice: Is it inherently a mixed method?. Research in the Schools, 17(2). Kretschmer, S., Langfeldt, B., Herzig, C., & Krikser, T. (2021). The organic mindset: Insights from a mixed methods grounded theory (MM-GT) study into organic food systems. Sustainability, 13(9), 4724. https://doi.org/10.3390/su13094724 Krueger, R. B., Sweetman, M. M., Martin, M., & Cappaert, T. A. (2020). Self-Reflection as a support to evidence-based practice: A grounded theory exploration. Occupational Therapy in Health Care, 34(4), 320–350. https://doi.org/10.1080/0738057 7.2020.1815929 Machleid, F., Kaczmarczyk, R., Johann, D., Balčiunas, ¯ J., Atienza-Carbonell, B., von Maltzahn, F., & Mosch, L. (2020). Perceptions of digital health education among European medical students: Mixed methods survey. Journal of Medical Internet Research, 22(8), e19827. https://doi.org/10.2196/ 19827 O’Cathain, A., Murphy, E., & Nicholl, J. (2008). The quality of mixed methods studies in health services research. Journal of Health Services Research and Policy, 13(2), 92–98. https://doi.org/10.1258/ jhsrp.2007.007074 Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in the Schools, 13, 48–63. Onwuegbuzie, A. J., & Johnson, R. B. (2021). The user’s guide to mixed methods analysis. Routledge. Palma, O. M., Díaz-Puente, J. M., & Yagüe, J.L. (2020). The role of coffee organizations as agents

MIXED METHODS-GROUNDED THEORY

of rural governance: Evidence from western Honduras. Land, 9(11), 431. https://doi.org/10.3390/ land9110431 Perez, A., Howell Smith, M. C., Babchuk, W. A., & Lynch-O’Brien, L. I. (2022). Advancing quality standards in mixed methods research: Extending the legitimation typology. Journal of Mixed Methods Research. https://doi.org/10.1177/ 15586898221093872 Puffer, K. A., & Pence, K. G. (2020). Behind dislike: adaptive purposes for undergraduates’ negative emotions in the career decision-making process. Sustainability, 12(19), 8071. https://doi.org/ doi:10.3390/su12198071 Sena Moore, K., & Hanson-Abromeit, D. (2018). Feasibility of the Musical Contour Regulation Facilitation (MCRF) intervention for preschooler emotion regulation development: A mixed methods study. Journal of Music Therapy, 55(4), 408– 438. https://doi.org/10.1093/jmt/thy014 Shim, M., Johnson, B., Bradt, J., & Gasson, S. (2021). A mixed methods–grounded theory design for producing more refined theoretical models. Journal of Mixed Methods Research, 15(1), 61–86. https://doi.org/10.1177/1558689820932311 Trull-Oliva, C., & Soler-Masó, P. (2021). The opinion of young people who have committed violent child-to-parent crimes on factors that enhance and limit youth empowerment. Children and Youth Services Review, 120, 105756. https://doi. org/10.1016/j.childyouth.2020.105756

303

Tsortanidou, X., Daradoumis, T., & Barberá, E. (2020). Waldorf inspired hyper-imaginative learning trajectories: developing new media literacies in elementary education. Early Child Development and Care, 191(7–8), 1287–1301. https://doi.org/10.1080/ 03004430.2020 Walsh, I. (2015). Using quantitative data in mixeddesign grounded theory studies: an enhanced path to formal grounded theory in information systems. European Journal of Information Systems, 24(5), 531–557. https://doi.org/10.1057/ejis.2014.23 Walsh, I., (2014). A strategic path to study IT use through users’ IT culture and IT needs: A mixedmethod grounded theory. The Journal of Strategic Information Systems, 23, 146–173. https://doi. org/10.1016/j.jsis.2013.06.001 Walsh, I. (2020). A combined variance and process perspective on IT culture and IT usage leading to the concept of IT acculturation. Systemes d’Information et Management, 25(2), 33–72. https://doi.org/10.3917/sim.202.0033 Wiess, C., & Bensimon, M. (2020). Group music therapy with uprooted teenagers: The Importance of structure. Nordic Journal of Music Therapy, 29(2), 174–189. https://doi.org/10.1080/0809813 1.2019.1695281 Wood, M. A. (2020). Policing’s ‘meme strategy’: understanding the rise of police social media engagement work. Current Issues in Criminal Justice, 32(1), 40–58. https://doi.org/10.1080/10345 329.2019.1658695

APPENDIX A: COMPLETE CODE TABLE FOR MM-GT STUDIES

304

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Moving Beyond Tradition: The Need for Expanded and Culturally Relevant Mixed Methods Design Typologies: Section 3 Conclusions J e s s i c a T. D e C u i r - G u n b y a n d P e g g y S h a n n o n - B a k e r Section 3 of the Handbook featured an expansion of traditional mixed methods research design typologies, focusing on the incorporation of cultural influences and design combinations. In compiling the section, as section editors, we faced a variety of challenges, particularly surrounding the organization of the chapters. Several chapters focused mainly on cultural issues or the customs, values, and beliefs of groups (Chapter 13, Douglas; Chapter 15, Rawlins et al.; Chapter 16, Jordan & Bartel) while others focused almost exclusively on discussing the expansion of typologies (Chapter 17, Mannell & Prost; Chapter 18, Mayoh et  al.; Chapter 19, Cook & Kamalodeen; Chapter 20, Scherman & Zimmerman; Chapter 21, Howell et  al.). Only one chapter emphasized the integration of culture into their design approaches (Chapter 14, Taghipoorreyneh). However, this is not unique to our section, but rather a reflection of the larger field of mixed methods research. There are many typologies in the mixed methods research field, including how to define this methodology (e.g., Johnson et al., 2007), types of integration (e.g., Creswell & Plano Clark, 2018), and rationales for using mixed methods research (e.g., Greene et  al., 1989). Although culture can be incorporated, most of these mixed methods research typologies do not explicitly address issues of culture. The transformative-emancipation

paradigm (Mertens, 2003, 2012) is one of the few frameworks in canonical mixed methods research that explicitly addresses culture and the role that culture plays within a mixed methods study. With the growing focus on cultural diversity throughout the world accompanied by complex problems and contexts, mixed methods research will need more advanced typologies, including those that incorporate culture. In order to move the field forward, it is imperative that we address this significant need. How can we continue to expand design typologies in mixed methods research? How can we ensure that those typologies are culturally relevant? How can we promote culturally relevant mixed methods research?

EXPANDING DESIGN TYPOLOGIES IN MIXED METHODS RESEARCH For decades, prevalence in the literature would suggest that the field of mixed methods research has been guided by the design typologies described by Creswell and Plano Clark (2007, 2011, 2018; see also Chapter 2, this volume), including explanatory sequential, exploratory sequential, convergent parallel, embedded (e.g., qualitative data embedded within quantitative data), and multiphase designs. These designs are seen as the canon, the essential

306

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

designs that the majority of MMR studies follow. However, it is difficult to stay within the confines of these designs while exploring complex phenomena or examining complicated relationships and contexts. This is one of the many reasons why there has been an expansion of the core typologies. Some expansion typologies include action research mixed methods designs (Ivankova & Wingo, 2018), mixed methods phenomenological research (Mayoh & Onwuegbuzie, 2015), fully integrated mixed methods designs (Creamer, 2017) and case study mixed methods designs (Cook & Kamalodeen, 2020; Guetterman & Fetters, 2018), as well as many others. The expanded typologies allow for more advanced design configurations. There needs to be continued typology expansion and a consideration in expanding the canon to include some of these newer typologies. This will allow researchers to collect and analyze more complex data within a concise framework. As previously mentioned, one area of growth in typologies has been in the area of qualitative-dominant mixed methods designs (cf. Poth & ShannonBaker, 2022). Hesse-Biber (2010) described the strength of using qualitative-dominant mixed methods designs including increasing representativity and generalizability of research, enhancing validity and reliability, as well as providing convergence in findings or data triangulation. This section of the Handbook featured several qualitative-dominant chapters focusing on grounded theory (Chapter 21, Howell et  al.), case study (Chapter 19, Cook & Kamalodeen) and phenomenology (Chapter 18, Mayoh et  al.), several core designs in qualitative research. It is important to expand upon this foundation by considering other qualitative approaches such as narrative inquiry, ethnography, discourse analysis, participant observation, and visual methods, among others. Although we advocate for the expansion of mixed methods research typologies, we acknowledge that doing so is a difficult task. Not all research methods are easily made compatible with other research methods. It is often difficult to combine many qualitative approaches with quantitative approaches because of philosophical differences. For instance, many quantitative methods are conceptualized from a positivist or post-positivist approach (e.g., there is one truth), while some qualitative approaches may be conceptualized from a constructivist or postmodern perspective (e.g., there are multiple truths) (Shannon-Baker, 2023). With diametrically opposite philosophical foundations, this would make reconciling the two approaches difficult or nearly impossible. Thus, the lack of advanced typologies, including the use of qualitative-dominant designs, often results from difficulties in negotiating or navigating philosophical tensions between combined designs.

CULTURALLY RELEVANT MIXED METHODS DESIGN TYPOLOGIES AND RESEARCH An area of need within MMR is the focus on the study of culture. There need to be more mixed methods studies that explore various areas regarding culture; for further examples, see the Conclusion to Section 5. In order to do so, MMR needs more culturally relevant typologies and designs that center on the experiences and from the perspectives of marginalized communities (DeCuir-Gunby & Schutz, 2014; Chapter 27, this volume). We recommend two approaches to address this need. The first approach is to modify current typologies to explicitly incorporate culture. This will allow researchers more flexibility in how typologies are used and interpreted. The second approach is to create culturally relevant typologies that are intended to address culture-related topics. This will enable researchers to specifically address culture-related topics within frameworks that readily consider cultural contexts, frameworks, and understandings. Although several of the authors in this section discussed issues of culture, only one of them took a critical stance in their work. Douglas (Chapter 13) focused on intersectionality. However, there was little focus on critical issues such as race, gender, or sexuality. Critical mixed methods design typologies would enable researchers to address issues of power and critique social structures within a variety of fields. There needs to be more development of how to incorporate criticality throughout the mixed methods research process and in this field. This can be done using both qualitative- and quantitative-dominant approaches. For instance, qualitative-dominant approaches could consist of using queer theory (Shannon-Baker, 2022) or a critical race approach (DeCuir-Gunby, 2020). Combined approaches with QuantCrit (Gillborn et  al., 2018) or critical quantitative methods (Zuberi & Bonilla-Silva, 2008) would be a way to take more of a quantitative-dominant approach. Critical mixed methods research design typologies would broaden the types of approaches that are taken in mixed methods studies.

MOVING BEYOND The mixed methods research community operates in an isolated space and is not fully connected to many academic disciplines. More mixed methods research published in the social sciences needs to draw from the methodological literature from this field. Likewise, it is possible that there are more

SECTION 3 CONCLUSIONS

expansions of mixed methods research design typologies as well as the incorporation of culture within these academic disciplines. As such, there needs to be a continual education of audiences on mixed methods research and a broader outreach to other academic disciplines. The mixed methods research community cannot assume that everyone knows this work. It is important for the mixed methods research community to help researchers to ground their work within the MMR lexicon, including the typologies. In addition, all mixed methods researchers need to have a nuanced understanding of their research population to best ground their work culturally and methodologically, as well as a clear sense of the audience to best present their research. This involves understanding mixed methods research design typologies and the cultural contexts in which the work is conducted.

REFERENCES Cook, L. D., & Kamalodeen, V. J. (2020). Combining mixed methods and case study research (MM+CSR) to give mixed methods case study designs. Caribbean Journal of Mixed Methods Research, 1(1), 47–76. Creamer, E. G. (2017). An introduction to fully integrated mixed methods research. Sage. Creswell, J. & Plano Clark, V. (2007). Designing and conducting mixed methods research. Sage. Creswell, J. & Plano Clark, V. (2011). Designing and conducting mixed methods research. (2nd ed). Sage. Creswell, J. & Plano Clark, V. (2018). Designing and conducting mixed methods research. (3rd ed). Sage. DeCuir-Gunby, J. T. (2020). Using critical race mixed methodology to explore the experiences of African Americans in education. Educational Psychologist, 55(4), 244–255. https://doi.org/10.1080/00461520. 2020.1793762 DeCuir-Gunby, J. T., & Schutz, P. A. (2014). Researching race within educational psychology contexts. Educational Psychologist, 49(4), 244–260. https:// doi.org/10.1080/00461520.2014.957828 Gillborn, D., Warmington, P., & Demack, S. (2018). QuantCrit: education, policy, ‘Big Data’ and principles for a critical race theory of statistics. Race Ethnicity and Education, 21(2), 158–179. https:// doi.org/10.1080/13613324.2017.1377417 Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method

307

evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https://doi.org/10.3102/ 01623737011003255 Guetterman, T. C., & Fetters, M. D. (2018). Two methodological approaches to the integration of mixed methods and case study designs: A systematic review. American Behavioral Scientist, 62(7), 900–918. https://doi.org/10.1177/0002764218772641 Hesse-Biber, S. (2010). Qualitative approaches to mixed methods practice. Qualitative inquiry, 16(6), 455-468. https://doi.org/10.1177/1077800410 364611 Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/15586898 06298224 Ivankova, N., & Wingo, N. (2018). Applying mixed methods in action research: Methodological potentials and advantages. American Behavioral Scientist, 62(7), 978–997. https://doi.org/10.1177/ 0002764218772673 Mayoh, J. & Onwuegbuzie, A. J. (2015). Toward a conceptualization of mixed methods phenomenological research. Journal of Mixed Methods Research, 9(1), 91–107. https://doi.org/10.1177/ 1558689813505358 Mertens, D. M. (2003). Mixed methods and the politics of human research: The transformative-emancipatory perspective. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 135–166). Sage. Mertens, D. M. (2012). Transformative mixed methods: Addressing inequities. American Behavioral Scientist, 56(6), 802–813. https://doi.org/10.1177/ 0002764211433797 Poth, C. N. & Shannon-Baker, P. (2022). State of the methods: Leveraging design possibilities of qualitatively oriented mixed methods research. International Journal of Qualitative Methods, 21, 1–11. https://doi.org/10.1177/16094069221115302 Shannon-Baker, P. (2022). Queering mixed methods research. In K. K. Strunk & S. Anne Shelton (Eds.), Encyclopedia of queer studies in education (pp. 589–594). Brill. https://doi.org/10.1163/978900 4506725 Shannon-Baker, P. (2023). Philosophical underpinnings of mixed methods research in education. In R. J. Tierney, F. Rizvi, & K. Erkican (Eds.), International encyclopedia of education (4th ed.), 12, 380–389. Elsevier. Zuberi, T. & Bonilla-Silva, E. (Eds.). (2008). White logic, white methods: Racism and methodology. Rowman & Littlefield.

This page intentionally left blank

SECTION 4

Designing Innovative Integrations with Technology

This page intentionally left blank

Expanding Innovative Integrations with Technology: Section 4 Introduction Timothy C. Guetterman

Have you considered how technology can help to achieve integration in mixed methods research? The following chapters provide practical insight into how to leverage software and technology for integration. Section 4 of the Handbook advances innovations in integration in mixed methods designs. Integration refers to the joining and interdependence of qualitative and quantitative research in mixed methods research (Bazeley, 2018). Our goal with this section was to help readers with the ongoing challenge of operationalizing integration. We sought to bring together a collection of chapters where authors would write about their actual experiences that would then provide important guidance with illustrative examples. In preparing Section 4, we asked contributors to write chapters with a pragmatic integration focus as opposed to conceptual discussions of integration. In other words, rather than only including theoretical possibilities, we encouraged illustrative examples accompanied with details about where and how integration is happening. What happened next was a surprise. As we worked on revising the chapters, I noticed an interesting thread emerging across the chapters related to technology. The role of technology manifests across the chapters in four major innovations addressing the following questions: How can software facilitate integration? How has visualization

emerged as a helpful method for mixed methods integration? What new opportunities for integration arise with the use of secondary data? How can game-based mixed methods research expand the concept of integration during data collection and analysis? Together, the chapters make a significant contribution to the field by demonstrating how software, technology and technology-oriented research bring additional ways to mine data and to mix data. The chapters included in Section 4 (see Table S4.1) advance new possibilities for innovative integrations with technology in mixed methods research designs. The aim of this Introduction is to pique your interest in integration and help you find the chapters that will be most useful. I discuss each innovation and how the chapters make a unique contribution to advance the field of mixed methods research.

HOW CAN SOFTWARE FACILITATE INTEGRATION? Chapters 22 (Kuckartz & Rädiker) and 23 (Inaba & Kakai) introduce the value of software for integrative analysis. Mixed methods researchers have used software to manage data in the qualitative strand of the study, but as I have previously

312

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table S4.1  Summary of Section 4 chapters: Designing Innovative Integrations with Technology Chapter authors (country affiliation)

Chapter title

Udo Kuckartz and Stefan Rädiker (Germany)

Using Software for Innovative Integration in Mixed Methods Research: Joint Displays, Insights and Inferences with MAXQDA Grounded Text Mining Approach: An Integration Strategy of Grounded Theory and Textual Data Mining A “Mixed Methods Way of Thinking” in Game-based Research Integrations Integrating Secondary Data from Ethnically and Racially Minoritized Groups in Mixed Methods Research Beyond the Joint Display in Mixed Methods Convergent Designs: A Case-Oriented Merged Analysis

Mitsuyuki Inaba and Hisako Kakai (Japan) Lisbeth M. Brevik (Norway) Daphne C. Watkins and Natasha C. Johnson (USA) Carolina Bustamante (USA)

argued, the use of software is not as common relative to quantitative research and statistical analysis software (Guetterman et al., 2015a). Over the past decade, qualitative data analysis software has evolved to include mixed methods features, such as the ability to store variables, to compare qualitative and quantitative results, and to examine patterns and relationships among themes and statistics for a particular dataset. Software with mixed methods features include Dedoose, Hyper­ Research, MAXQDA, NVivo and QDAMiner. I argue that software has the potential to enable new and efficient integrative analyses that otherwise require manual compilation. Software is mentioned throughout many chapters of this Handbook; two chapters in this section make unique contributions through their descriptions. Whereas Chapter 22, Kuckarz and Rädiker describes MAXQDA software for mixed methods research, Chapter 23, Inaba and Kakai, describes software for grounded theory text mining. These two chapters move far beyond software for the management of qualitative data in mixed methods research and describe how software can enable innovative integration strategies. Both chapters unlock possibilities for integration through software technology and provide helpful guidance for readers. In Chapter 22, Kuckarz and Rädiker differentiate the application of MAXQDA software in four forms of integration: treating qualitative codes as variables to integrate with quantitative data; transforming qualitative data and using statistics; constructing joint displays; and achieving full integration by systematically compiling analyses, joint displays and meta-inferences. In Chapter 23, Inaba and Kakai provide a description and example of grounded text mining (GTxA). While mixed methods research traditionally involves the collection, analysis and integration of qualitative and

quantitative data, their innovative approach uses a single text dataset that is analyzed in an integrative manner. They advance a series of steps for GTxA: (1) using text mining software; (2) conducting constructivist grounded theory coding; (3) autocoding of those codes from step 2 through text mining and quantitizing the codes; and (4) examining the results in an iterative way to gain a better understanding.

HOW HAS VISUALIZATION EMERGED AS A HELPFUL METHOD FOR MIXED METHODS INTEGRATION? The role of visualization in integrative analysis is one of the greatest advancements in mixed methods research in the last decade. Visualization is a way to represent data and to facilitate integration through techniques such as joint displays that represent the integration of qualitative and quantitative research (Guetterman et  al., 2015b). Across four chapters in this section, visualization was an integral part of mixed methods analysis. In Chapter 26, Bustamante’s case-oriented visual displays and joint display of integrated results, she incorporates a theoretical perspective. In Chapter 24, Brevik’s joint displays indicate integrated analysis across phases of research. From a qualitative perspective, Brevik’s inclusion of a game screen capture image further brings her topic to life for the reader that would be difficult to conjure through text only. Also in Chapter 22, Kuckartz and Rädiker discuss the use of joint displays for an innovative integrative analysis. They argue that joint displays are made possible through specialized software and substantially harder to conceive without. In Chapter 23, Inaba and Kakai also include numerous visualizations in

SECTION 4 INTRODUCTION

their demonstration of GTxA. They identify Figure 23.9 in their chapter as a joint display of the integrated results of grounded theory analysis and autocoding and discuss how the joint display informed implications of their research. Interestingly, they also include a visualization from software-produced correspondence analysis of focus group data from students and professors to indicate areas of commonality and difference. I would characterize their correspondence analysis as one of the earliest types of joint displays (Guetterman et  al., 2021). These chapters provide essential guidance for how to use visualizations effectively in MMR.

WHAT NEW OPPORTUNITIES FOR INTEGRATION ARISE WITH THE USE OF SECONDARY DATA? The use of secondary data often requires software given the sheer volume in the case of big data. Two chapters use secondary data in different ways. In Chapter 25, Watkins and Johnson focus on integrating secondary data, which can include large data sources or smaller files. As they explain, the expansion of technology has improved data collection, management and analysis, making existing data more accessible and usable for secondary purposes. Including secondary data opens newer data sources for conducting mixed methods research and analysis. For example, they address the problem of underrepresentation of ethnically and racially minoritized groups in research, and how conducting mixed methods research with secondary data could overcome concerns regarding small sample sizes, lack of funding and missed opportunities. While big data brings opportunities for mixed methods, challenges also arise. In Chapter 23, Inaba and Kakai mention the use of big data, and their use of text mining software could extend to big qualitative data sets. However, they cite a challenge in applying their GTxA procedure because the grounded theory aspects require reading the entire text, which is prohibitive for big data. Perhaps, future innovations could address these concerns through representative sampling.

HOW CAN GAME-BASED RESEARCH EXPAND THE CONCEPT OF INTEGRATION DURING DATA COLLECTION AND ANALYSIS? Game-based research is often complex and interdisciplinary involving fields such as education,

313

information technology, computer science, human factors psychology and content areas in the case of educational games. In Chapter 24, Brevik presents the interplay of gaming and mixed methods research and describes ample opportunities for mixed methods data collection and analysis throughout. Brevik situates the purposes for integration in game-based research in Poth’s (2018) typology of purposes (corroboration, completion, explanation, development, infusion, and innovation). She then applies ten integration strategies to a study of game play among teenagers in Norway. Although Brevik’s study focused on game play, the ten strategies are likely applicable to other technology-oriented contexts such as studying the use of software, interaction with technology like health-related devices, and how people obtain services and information through technology. In this regard, game-based research provides an excellent example of the complexity of technology-related topics that mixed methods research can elucidate.

CONCLUSION Integration is central to mixed methods research designs, and the chapters in this section present considerable innovations in conceptualizing and conducting integration. In particular, chapters present innovations related to software usage, visualizations, secondary data and game-based integration with implications for complex designs. Developing a mixed methods research design requires careful attention to integration. As noted in these chapters, the process of integration is often creative rather than following a manualized approach. However, what is clear across these chapters is the authors’ commitment to a systematic process of integration that is well thought out and well articulated. The examples of integration in these chapters are meant to help readers in guiding their own mixed methods research practice. I encourage you to think about which of these innovations may be transferrable to your own research.

REFERENCES Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage. Greene, J. C. (2007). Mixed methods in social inquiry. Jossey-Bass.

314

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Guetterman, T. C., Creswell, J. W., & Kuckartz, U. (2015a). Using joint displays and MAXQDA software to represent the results of mixed methods research. In M. McCrudden, G. Schraw, & C. Buckendahl (Eds.), Use of visual displays in research and testing: Coding, interpreting, and reporting data (pp. 145–176). Information Age. Guetterman, T. C., Fàbregues, S., & Sakakibara, R. (2021). Visuals in joint displays to represent integration in mixed methods research: A methodological review.

Methods in Psychology, 5, 100080. https://doi.org/ 10.1016/j.metip.2021.100080 Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015b). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. The Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm.1865 Poth, C. N. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage.

22 Using Software for Innovative Integration in Mixed Methods Research: Joint Displays, Insights and Inferences with MAXQDA Udo Kuckartz and Stefan Rädiker

INTRODUCTION It is standard practice that quantitative data is analyzed with the help of specialized software. Somewhat less natural, but common nowadays, is to analyze qualitative data also with the help of appropriate software. But what about mixed methods projects that deal with qualitative and quantitative data? Can special computer programs be used to conduct an integrative analysis of qualitative and quantitative data and results? How can specialized software help you to compile, visualize and jointly analyze the different data and results on a case-by-case or group basis? In this chapter, we show how the software program MAXQDA can be used to apply integration strategies in mixed methods studies. In doing so, we focus on the aspects of interactivity, which come into play in different ways when analyzing mixed data with MAXQDA. For example, in visualizations and results tables, the original data behind the results can be accessed, allowing you to switch between quantitative and qualitative data analyses at any time. MAXQDA not only enables the interactive analysis, the joint display and the presentation of mixed methods data, but it also provides a good overview of complex material and ensures transparency, and improves the quality of the analysis process (Kuckartz &

Rädiker, 2019, 2021, 2022). Of course, many authors have already worked on integration strategies (e.g., Bazeley, 2012, 2018; Creswell & Plano Clark, 2018; Hitchcock & Onwuegbuzie, 2022; Kuckartz, 2017; Onwuegbuzie & Johnson, 2021) and proposed strategies such as qualitizing and quantitizing (e.g., Onwuegbuzie & Leech, 2019, 2021; Sandelowski et al., 2009; van Velzen, 2018; Vogl, 2019). When we write about innovative integration strategies in this chapter, we are focusing on innovative applications using software specifically designed for this purpose. These can also involve the implementation of forms of integration already described in the literature. This chapter is organized as follows: first, we introduce different types of integration, which mainly depend on the kind of realized samples and the mixed methods design used. In the main part of the chapter, we present four forms of integration: (1) treating codes as categorial variables; (2) quantizing and using statistics; (3) combining quantitative and qualitative data and results in joint displays; and (4) full integration by compiling analyses, joint displays, meta-inferences and insights. For each form, we explain the usage of MAXQDA and discuss the role interactivity plays in that form. Finally, we give a brief outlook on where further developments can be expected regarding software support for mixed methods integration.

316

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

STRATEGIES FOR INTEGRATION AND THE ROLE OF SOFTWARE Integration is the challenge, but often integration does not take place in mixed methods research or it plays only a minor role. In this contribution, we focus on integration in the process of analysis. Of course, integration can also happen in other stages of a mixed methods project or even in all phases of the research process, as Schoonenboom and Johnson (2017) are describing in their so-called “fully integrated mixed designs”. However, the problem we see with such designs is that confusion is created by a multiplicity of points of integration and that no clear distinction between qualitative and quantitative methods takes place anymore. In the following, we will therefore start from the widely accepted differentiation of mixed methods designs into the three basic design types proposed by Creswell and Plano Clark (2018): the convergent, the explanatory sequential and the exploratory sequential design. For a discussion of the evolution of these designs, see also Chapter 2 (this volume). Furthermore, we assume in the following that there are always qualitative and quantitative parts in a mixed methods project. Of course, there may be different forms of data collection in both fields, such as narrative interviews, focus groups and observations in the qualitative branch. The more complex the designs, the more complex the analysis and the more difficult the integration. For a discussion of a visualizing approach to help manage the complexity, see also Chapter 8 (this volume). Our contribution starts from the simple case of a qualitative and a quantitative study with one data type in each branch. The examples presented can be further developed and also transferred in the context of more complex designs. In our previous works (e.g., Kuckartz & Rädiker, 2021), we dealt with different strategies of integration and argued a distinction between four main types of integration: • In results-based integration, the integration step takes places after the analyses of the qualitative and the quantitative strands have been completed (at least partially) and the results of both strands are combined, compared and related to each other. • Data-based integration means that researchers relate and jointly analyze qualitative and quantitative data during the analysis phase. Therefore, qualitative and quantitative data are needed for the same cases and it must be possible to clearly assign both types of data to individual cases.

• By transformation-based integration, we mean that qualitative data is transformed into quantitative data or, conversely, quantitative data is transformed into qualitative data. Integration then takes place in one of the two “data worlds”. • Sequential integration takes place when the results of one strand are used to conduct the other—for example, for the development of data collection instruments or sampling purposes. In the context of integration, software plays an important role, but in different ways for each strategy type. In one transformation-based strategy, the qualitative data are converted into quantitative ones—for example, by transforming the frequency of qualitative code occurrences into variable values on a case-by-case basis. Joint analyses then take place only within the quantitative world. Statistical analyses are carried out in which the data of both strands are included and merged; their results are presented in a report. All this is impossible to imagine without software. Sequential mixed methods designs are also hardly feasible without software. Here, a crucial point is the construction of the respective second study on the basis of the results of the study conducted first. For example, if the construction of the questionnaire for a survey is based on the results of a qualitative study, it makes sense to do this construction in a planned and systematic way. For this purpose, it is necessary to indicate in the results of the qualitative study the sources for the questions and standardized answers of the survey. Corresponding functions of the software have to allow the realization of such linkages. In addition to this point of integration at the transition from the qualitative to the quantitative study (and vice versa in explanatory designs), there is another point of integration in sequential designs after the completion of both research branches. At that point, the results of both studies can be integrated in several ways. Whether and in which form this happens, however, depends on the relevance attributed to the two strands. If, for example, in exploratory designs the qualitative study is considered only as a preliminary study for the construction of the survey, the interest in it is likely to be rather low; it is then the results of the survey that are attributed the greatest relevance. In all mixed methods projects, software plays an important role; without it, the different integration variants are hardly realizable. This is even more true for the other two types of integration strategies, the results-based and the data-based strategies. Databased strategies in particular open up a wide range of possibilities because both qualitative and quantitative data are available for the same case units.

USING SOFTWARE FOR INNOVATIVE INTEGRATION IN MIXED METHODS RESEARCH

A case unit can be an individual as well as a family, institution, group or similar, and the software must be able to handle and support the analysis of qualitative and quantitative data for each case and connect them on a case-by-case basis. In general, software has a twofold function in a mixed methods project. It is the central tool of diagnosis during the analysis phase—for example, case comparison tables created with the help of software can be used to discover similarities and differences between case units and to identify patterns. The second no less important function is the presentation function. Findings can be organized with the help of software in such a way that they represent important elements of a publication or conference poster. This is especially true for the joint displays presented below in the section “Integration by Way of Joint Displays”. Considering all these aspects, a software program for mixed methods analysis should provide at least the following features (cf. also Kuckartz & Rädiker, 2023): • import qualitative and quantitative data and allow their connection on a case-by-case level; • handle several qualitative data sources from the same or different samples (interviews, focus groups, observations, documents); • handle several quantitative data formats (Excel, SPSS, etc.); • link segments of data and/or of results; • text search and automatic coding for fast access to data and results; • typical analytical techniques for qualitative data, particularly coding and writing memos; • typical analytical techniques for quantitative data, particularly descriptive statistics, aggregation of variables, and scale construction. MAXQDA is a software tool for qualitative and mixed methods data analysis that offers all these features (www.maxqda.com; Kuckartz & Rädiker, 2019). In addition, and beyond the dual function of being a tool of diagnosis and a means of presentation mentioned above, MAXQDA offers another very important feature for the analysis process—namely, interactivity. In contrast to a static display, MAXQDA allows one to work with mixed methods data, results tables and visualization in an interactive way—for example: Browse topics and themes. In interactive tables, one can inspect different themes one after the other with a single mouse click; on paper, one is bound to the paper size, while on screen, one can scroll

317

and navigate, browse and focus on cells, rows, columns of a table, etc. Go to original data. In visualizations and in result tables, MAXQDA retains the link to the original data, such as coded text passages, so that they can be accessed with just a few mouse clicks. Inspect context. This linkage to the original data, allows one to inspect the context, for example, of a coded text segment, at any time for confirming or checking results instead of only relying on aggregated data. Work with annotations. One can add new, or read existing, notes on the original data and on results displayed in tables and visual outputs, etc. These annotations can be accessed from several locations. Select cases. The linkage of qualitative and quantitative data makes it possible to select and access cases of the other data world from within a results table or visualization. Adjust view. In the tables and visualizations, one can highlight values, zoom in or out, move and hide columns. (Rädiker, 2018) With the help of appropriate software, such as MAXQDA, researchers are able to analyze complex data interactively and rigorously, either alone or in a team, while ensuring transparency. In this sense, innovation means finding new ways for analyzing complexity.

FOUR FORMS OF INNOVATIVE INTEGRATION In this section, we will present different types of innovative integration and describe which innovative strategies have been developed for MAXQDA, using some examples from research. In doing so, we refer to four different forms of integration. The first form of innovative integration is to treat the codes of the qualitative study as categorial variables and to perform integrative analyses with the variables of the quantitative data. In doing so, procedures of combinatorics and statistics for categorical data are applied—both, of course, using appropriate software. We present an example of this type of integration, which is innovative mainly due to the increased degree of interactivity. The second form uses transformation: qualitative data are quantified, so that integrative analyses take place only in the quantitative data world—i.e., with the help of statistical analysis tools. Various authors have already presented elaborated approaches—for example, the use of

318

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

correspondence analysis, multidimensional scaling, cluster analysis and model building. The third form is to construct joint displays: in this variant, integration takes place neither by categorization and combinatorics, nor by quantification/transformation, but by means of displays in which the data or results of both strands are contained and combined. Such displays are usually referred to as joint displays or integrative displays. As an example, we present here an innovative Case Comparison Table created flexibly and interactively with MAXQDA software. Fourth, we present an innovation that affects the entire analysis process: the full integration of all quantitative and qualitative results related to a research question in a single location. As an overarching innovative way of integration, we describe the tool QTT (questions, themes and theories) of the MAXQDA software.

Integration by Treating Codes as Categorial Variables A frequently found qual–quan combination in mixed methods research is the combination of qualitative interviews and/or focus groups with a standardized questionnaire. The analysis of the qualitative interviews is done in such a way that the interviews are systematically worked through. In the course of the analysis, codes are formed and text passages are coded, resulting in the identification of topics and themes. Recently, algorithmic methods such as factor analysis or topic modelling have been increasingly used to identify themes and topics in the qualitative data (e.g. Van Haneghan, 2021). In the following, however, we focus on manual coding. This is a central activity of data analysts in the thematic analysis method and in qualitative content analysis (Boyatzis, 1998; Guest et al., 2012; Kuckartz, 2014; Schreier, 2012). Coding of qualitative data also plays an important role in approaches that follow grounded theory methodology, but coding has a slightly different meaning in grounded theory (Kelle, 2007). When the data is coded using the qualitative content analysis or thematic analysis method, categories and subcategories emerge. The software stores the information about which text passages are assigned to which categories and how often this is the case. Further information about the coded segments is also stored and is available for further analyses—e.g., the extent of the respective coded text passages in words and the words that occur in the coded text passages. In a study of climate change perceptions and personal behaviour, it is known from the coding whether and how

often an interviewee talked about their mobility behaviour during the interview, whether they ride a bicycle to work, whether they take family vacations by plane, and whether they have purchased a fuel-efficient car or an electric vehicle in recent years. Also coded, for example, are a person’s perceptions of climate change, whether they plan to change their behaviour and whether they have made sustainable financial investments. In the process of analyzing the qualitative data, the categories have the function of opening up and organizing the data, identifying and analyzing themes and key topics, and comparing different cases (interviewees) with each other. In order to integrate the results of the qualitative and quantitative strands in a mixed methods project, the categories and subcategories can also be viewed as dichotomous or as categorical variables: for a case, a specific category or subcategory (e.g., “sustainable investments”) was coded or not coded (0/1); or different types of mobility were recorded as subcategories (“bicycling”, “use of public transportation”). Qual–quan integration now takes place in such a way that the categories of the qualitative study are analyzed with regard to their relationships to the variables of the quantitative study. This is done using procedures of combinatorics and statistics for categorial data. When the categories are used in this way, we do not think it is quantitizing, because no transformation takes place; it is simply using the categories developed during the qualitative analysis in a different mode of analysis. The following examples show how this can happen. Fig. 22.1 shows a simple example of a statistical crosstab in which the qualitatively analyzed information as to whether the category “sustainable investments” was coded for a case (0/1) forms the rows and the quantitative variable “age group” forms two columns. The crosstab has been created with Stats, a module that is fully integrated in the qualitative data analysis software MAXQDA. Stats extends the possibilities for qualitative analysis and allows one to perform mixed methods data analysis in a quantitative way. It offers commonly used descriptive and inferential statistics, such as frequency tables, calculation of descriptive statistical measures, crosstabs, analysis of variance, correlation and scale analysis. In the dialog boxes of MAXQDA Stats (Figure 22.2), not only is the standardized data in the form of variables available, but also the categories created during qualitative analysis, which makes it easy to analyze the relationships between the quantitative and qualitative data. For analysis in Stats, the qualitative categories can serve as dependent criteria, as in the crosstab example in Figure 22.1 or as independent criteria, as in

USING SOFTWARE FOR INNOVATIVE INTEGRATION IN MIXED METHODS RESEARCH

319

Figure 22.1  Crosstab created with MAXQDA Stats with using a code as categorial variable

Figure 22.2  Selecting categories (codes) and variables in MAXQDA Stats for statistical analysis Figure 22.3, which shows the results of an ANOVA (analysis of variance). The category “responsibility” has three subcategories (high, medium and low) and is used as a grouping factor for the cases which are compared for the means of their ages. In MAXQDA, the treatment of code occurrences as categorial variables is not only possible in the module Stats, but can also be used in standard analysis tools and visualizations. Figure 22.4 shows a so-called Document Map in which the cases (documents) are shown in a two-dimensional display according to their code occurrences and variable values. The more similar the code occurrences and the variable values of two cases are, the closer together they are presented in the diagram. Multidimensional scaling is used to calculate the positions and a hierarchical cluster analysis is used to assign the same colour to similar cases. In Figure 22.5, we present a comparison of code occurrences for two different groups in a so-called Mixed Methods Crosstab. Each row contains a different subcategory that can be considered as a categorial variable, and the columns represent two groups formed by the values of a standardized variable (“Math as advanced placement course” = No/Yes). The cells provide information about the relative number of cases in each group for which the code was assigned.

While working with codes as categorial variables, it is important to consider the overall number of cases and the different group sizes. The use of categorial variables in statistical analysis depends on having a sufficient number of cases and is not valid for, say, five interviews, because with just a few cases it would not be correct to compare mean values or to compute percentages. However, it may be meaningful to create a visual Document Map with only five interviews and, conversely, one might run into a problem trying to visualize 200 cases on a small map. The smaller the sample, the more important it becomes to go back (and forth) from the results of the calculations to the original qualitative data to explore and confirm relations and differences found in the mixed methods analysis, and the more important it might be to aggregate small groups into a larger one. This leads to an important point we want to emphasize here: from our perspective, innovative integration in mixed methods analysis is mainly supported by interactive software tools. MAXQDA offers the following interactive features that are relevant in the examples presented above. • Several rows or columns in the result tables of MAXQDA Stats can be merged interactively—for example, different age groups can be combined

320

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 22.3  ANOVA table created with MAXQDA Stats using a code as the factor with three levels

Figure 22.4  Document Map in MAXQDA (partial view). The cases (interview respondents) are shown in a two-dimensional display using multi-dimensional scaling on codes as categorial variables and standardized variables into one group. The results table is updated immediately, no time-consuming manual recoding of variables, which is typical for statistical programs, is necessary. Similarly, calculated values, such as column percentages or standardized residuals, can be turned on and off and no recalculation of the table is necessary. • In Stats, the cases that “lie behind” a cell in a crosstab (Figure 22.1) can be saved as a group of cases for a subsequent qualitative review, which is particularly useful for exploring extreme cases or deepening the analysis of groups containing just a few cases.

• In the Document Map (Figure 22.4), different analysis modes can be switched on and off, and the cases of a group identified by the cluster analysis can also be saved as a cluster for further qualitative analysis. Clicking a case on the map allows one to open and read the corresponding qualitative data—for example, the interview transcript. • In Mixed Methods Crosstabs (Figure 22.5), a double-click on a cell lists all the corresponding coded qualitative data for further analysis, and the whole table can be displayed as an interactive view of the underlying coded text passages.

USING SOFTWARE FOR INNOVATIVE INTEGRATION IN MIXED METHODS RESEARCH

321

Figure 22.5  Mixed Methods Crosstab in MAXQDA. Multiple categorial variables (subcategories) form the rows, the values of a standardized variable form the columns

Integration by Quantitizing and Use of Statistics The second variant of integration in the phase of analysis (quantification and use of statistical methods) works with the quantification of the qualitative data. This quantitizing can be done in different ways (Sandelowski et  al., 2009). In qualitative interviews as a data collection method, the following type of quantitizing is widely used: the interview is first coded and then the frequency of occurrence of a code is transformed into a variable, as in the following example: an interviewed teacher talked about discipline difficulties with students four times in one interview. By quantitizing, a variable “discipline problems” is formed and this variable now gets the value “4”. Thus, the categorized verbal data is actually transformed into its numerical equivalent here. Unlike Sandelowski et  al. (2009), we argue for distinguishing categorizing from quantitizing. In our view, categorizing, as explained in the section above, “Integration by Treating Codes as Categorical Variables”, is not a kind of quantifying but an ordering process: things and/or states of affairs are compared with each other and similar things are assigned to the same category. For example, an object is assigned to the category “apple” or “pear”. Both are subcategories of the common super category “fruit.” In everyday life, categorizations take place all the time, even without our being aware of it. If we categorize an object as an apple, we involuntarily compare it with pears and other types of fruit. The process of categorization is preceded by a (qualitative) comparison, not a counting process or a conversion into a number. Only the latter we call quantitizing—for example, it is counted how often a code was assigned in an interview and this number can then be used for integrative analysis with the variables of the quantitative strand. By doing this, the qualitative data is transformed in a metric variable.

Many authors have explored the different options for quantitizing and then applied complex statistical methods for analysis of the transformed qualitative data. This was done, for instance, by Van Haneghan (2021), who used exploratory factor analysis, by Péladeau (2021), who used cluster analysis, and Dickinson (2021), who used correspondence analysis. If the frequencies of the codings are counted per code and stored as variable values per case, the resulting variables have interval levels. As a reminder, an interval scale is a metric scale in which there is not only a rank order, as in the ordinal scale, but also the distance between any two adjacent values is called an interval, and intervals are always equal. Variables of this type can be used in the well-known methods of inferential statistics—e.g., in regression analysis, factor analysis and correlation analysis. Often in mixed methods analyses, these variables, which are actually metric, are dichotomized—that is, only a distinction between 0 = “code occurs” versus 1 = “code does not occur” is used. This is how Cox et  al. (2021) proceeded, for example. The authors conducted research on the implementation of educational programs, including interviews with regional counsellors in their project. The information about implementation processes provided by these key informants in the interviews was transformed into a dichotomized variable (0 = no identified implementation problems, 1 = implementation challenges) and then used in multiple regression analyses. In the case of variables with 0/1 coding, it may look at first glance as if information has been transformed into numerical relations and as if categorizing and quantitizing are the same thing. But this is not true: if the existence of a category is transformed into the value “1” (e.g., “Catholic religion” = (1), no quantification takes place, instead of “1” any non-numeric character could be assigned. However, if I dichotomize a metric variable—for example, blood pressure beyond a

322

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

threshold (>140 = high)—the 0/1 variable hides a dichotomized continuum. The same happens when dichotomizing qualitative code frequencies, which has been performed by Hitchcock et al. (2021). They develop an innovative approach to use the method of “hierarchical linear modelling” (HLM) in the context of mixed methods data analysis. A major advantage of HLM is that it is possible to work with different levels of analysis (e.g., school, class, student). Furthermore, the quantitized variables can be included in the models both as independent and dependent variables and finally it is possible to include variables of different scale levels—metric, ordinal/categorial, or dichotomous. HLM is so far only offered in statistics software and not in QDA software. All preparatory steps of quantitizing can be performed with MAXQDA or with Stats, the statistics module of MAXQDA. The corresponding data matrices “cases by variables” can be exported in SPSS or Excel format, allowing smooth transition to statistical software. Statistical modelling places considerable demands on the data, particularly in terms of sample size. Thus, using the innovative idea of integrative modelling proposed by Hitchcock et al. is particularly useful when a large amount of qualitative data has been collected, such as from surveys, Twitter or YouTube.

Integration by Use of Joint Displays For the third way of innovative integrative analysis, neither categorization and combinatorics, nor quantitizing and statistics are required. Instead, data and/or results from both strands are presented

in a single diagram, commonly referred to as a joint display (Guetterman, 2015a and 2015b). Often, these joint displays are tables or matrices that present both kinds of data and results, but joint displays can also be created in the form of a visualization, such as a concept map or mind map. The saying “a picture is worth a thousand words” only applies to joint displays that actually take the form of pictures or graphics, but it is also true for many other joint displays. A combined display prototypically and obviously represents a linkage, an integration of both research strands, and makes it naturally clear to recipients that insights can be achieved through mixed methods research. Researchers also benefit from these displays as they provide various opportunities for a mixed analysis, provide a bridge for drawing conclusions and meta-inferences, and even the process of creating such a display usually deepens the analysis (Fetters & Guetterman, 2021; HaynesBrown & Fetters, 2018). “No mixed methods software, no joint displays”: this is how the relationship between software and joint displays can be summed up succinctly. Although some joint displays could be created using Microsoft Word or other word processing programs, the core of the statement is true: without the power of specialized software, joint displays are hardly conceivable. In Figure 22.6, we present an extract of a case comparison table with mixed data that has been created with MAXQDA. In this type of joint display, the cases form the rows, while qualitative categories and quantitative variables form the columns. In the example, for three interviews the coded text passages on “Societal influence” and “Learnability” have been summed up in the researcher’s own

Figure 22.6  Case comparison table with mixed data in MAXQDA (partial view)

USING SOFTWARE FOR INNOVATIVE INTEGRATION IN MIXED METHODS RESEARCH

words and are presented together with two standardized variables—the “Age group”, and a scale measuring climate awareness. In contrast to aggregating ways of analysis presented before (mean calculation, relative frequencies etc.), in this joint display, the analysis works on a case-level and the only aggregation that takes place is by writing summaries for the qualitative statements regarding the two topics “Societal influence” and “Learnability”. No transformation or statistical procedure is necessary. To create such a display, the software has to be able to link both the qualitative data and the quantitative data on a case-by-case level. The importance of a step-by-step development of a joint display is shown in the excellent description by Fetters and Guetterman (2021). In their example, the individual steps of the display creation process are described in great detail and it becomes clear that a joint display is not something that is fixed from the beginning and only has to be put down on paper, but that creating such a display is a process consisting of several cycles in which one tries to approach an optimal end result step-by-step. As with the sections before, we would like to emphasize the interactivity of the way of innovative integration we have presented. • The joint display in Figure 22.6 can be sorted by variable values, which is useful to identify patterns and differences between cases or groups of cases or to analyze the relationships between the qualitative and quantitative aspects studied. • Columns can be moved around to place different aspects close to each other, and columns can be hidden and redisplayed, which allows one to focus on specific aspects and to handle very large tables. • The table can be re-created with a few mouse clicks to add or update cases and columns, supporting an ongoing and stepwise process of optimization and refinement of the display.

Full Integration by Systematic Compilation of Analyses, Joint displays, Meta-inferences and Insights The fourth way of innovative integration (fully integrated synthesis) represents an overarching form of integration and, as such, is already an innovation. Mixed methods research is always addressing specific topics and specific questions;

323

often the goal is to generate hypotheses and theory grounded in the data. QDA software can be a valuable aid in this process—for example, for developing and applying coding frames when analyzing qualitative data. Thus, the data is sorted into a differentiated system of order, much like objects are sorted into a drawer cabinet. This systematic organization with the simultaneous development of a meaningful coding scheme is already an achievement in itself. In a mixed methods project, however, the question then arises of what to do with the sorted material in the next steps and how to integrate it with the data and results of the quantitative study. It is often the case that, especially in the analysis of qualitative data, a great deal of work is invested in coding, but for the later analysis steps and the integration with the results of the quantitative study, there is often not only a lack of time, but also a lack of concept, so that it is easy to lose the overview. In MAXQDA, the integrative tool QTT helps one to focus on the research questions or, if necessary, to develop them first. Technically, QTT works like a collector’s sheet, a repository, in which everything relevant to a research question is collated, on which questions are recorded, topics and key categories are focused, and answers to the research questions are formulated. Here, not only individual integrative analyses are created, as is the case with the case comparison table or joint displays, but, on a meta-level, all analyses carried out in the analysis process are compiled, commented on and interpreted. Following Onwuegbuzie and Johnson’s concept of fully integrated design, one could speak of a “fully integrated analytical display”. The organization of the QTT is similar to the organization of contacts in a smartphone. The workspace (Figure 22.7) is divided into different sections: • Subject: The subject of the research is described here. • Research question(s): Detailed description of the research questions and the terms used in the project, also mentioning the theories on which the research is based, if applicable. • Related codes and themes: All codes that are important for the research question(s), including their definitions. • Important segments: List of particularly important segments of the qualitative data, with the option to remove or add new segments. • Summary tables: Summary tables or case comparison tables for the research questions.

324

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

• Related memos: List of memos linked to the topic and relevant data. • Visuals: visualizations especially joint displays related to the topic. • Concept maps: Concept maps, infographics and other maps created in the analysis process. • Integration: conclusions from the qualitative and quantitative findings and integration of the insights. If necessary, further sections can be added, which then appear as tabs in the QTT. All elements of a QTT worksheet can be inserted directly from the corresponding functions. For example, if you create a concept map, you can insert it into a selected worksheet using the “Send to QTT” option. The same option is available for other visual tools and joint displays. The facility for directly entering comments and recording insights for each element in the

Figure 22.7  The QTT workspace of MAXQDA

QTT should be emphasized. The entire worksheet can be exported as a Word file in DOCX format, and parts of it can be transferred directly into a research report.

IMPLICATIONS AND FUTURE DIRECTIONS In this chapter, we have shown how specialized software can be used in mixed methods research with different integration strategies. For an additional example, see also Chapter 23 (this volume). The use of software in data analysis offers the possibility of better and more accurate results. This is evident for both quantitative and qualitative analysis. In quantitative research, hardly anyone would seriously suggest performing statistical analysis without a computer, few would use just a calculator and nobody would recommend pencil and paper. The use of special software also

USING SOFTWARE FOR INNOVATIVE INTEGRATION IN MIXED METHODS RESEARCH

offers immense advantages in the analysis of qualitative data. These can be well illustrated by the evaluative criteria mentioned in the discussion about the standards of qualitative research. Guba and Lincoln (1985) identified four general criteria: credibility, transferability, dependability and confirmability. These were supplemented by Miles and Huberman (1994, pp. 278–280) with additional criteria including auditability and authenticity. For each criterion, it can be demonstrated in detail how fruitful the use of software can be in the analysis of qualitative data. Let us take the criterion of confirmability as an example. This describes how and to what extent other researchers can agree with the findings. The aim is to justify the results and interpretations from the data clearly and plausibly, and not just to reflect the subjective opinion or perception of the inquirer. Special QDA software makes it possible to link findings to the original data at any time, even to the original sounds and images if audio and video recordings were made during interviews or focus groups. In this respect, the abovementioned criterion of authenticity is fulfilled in a much better way. QDA software also allows comprehensive secondary analyses through the exchange of complete project files—i.e. comparisons with the findings of other researchers are made possible. Mixed methods analysis is more complex than purely quantitative or purely qualitative analysis; it makes greater demands on both the skills of the analysts and the technology for its practical implementation. If in both traditional paradigms— quantitative research and qualitative research—the use of software proves to be so useful, even indispensable for some types of analysis, then logically this should be the case even more in mixed methods data analysis. In 2011, Creswell et al. developed best practices for mixed methods research in the health sciences and claimed that: High-quality mixed methods applications should [ … ] specify the equipment and expertise available to support sophisticated mixed methods research. Examples of these resources might include software packages that facilitate the relating of quantitative and qualitative data, expertise in developing quantitative instruments from qualitative findings, and expertise in mixed methods research designs and approaches. (2011, p. 24)

This chapter has shown how to facilitate the linking of quantitative and qualitative data with the MAXQDA software. Research needs ideas, innovation and creativity. This is especially true for research methods because they move us forward

325

on the way to the insights that we need for the transformation of society. Mixed methods research itself already represents a process of methodological innovation; it is especially dependent on further innovative and creative developments. In particular, it is integrative analysis that benefits from new ways and forms of analysis, displays and findings. Joint displays play a very important role in this process, and these depend on the use of special software. In this article, we have outlined the ways in which software can be used to provide deeper insights and better forms of presentation of research results. We have also described the ways in which methods developed in earlier practice have been incorporated into the MAXQDA software while a variety of integrative techniques have been newly developed. Further advances on this path of innovation will certainly continue; many empirical researchers are constantly engaged in the development of new techniques and the creative transfer of proven techniques from other fields to mixed methods analysis.

WHAT TO READ NEXT Kuckartz, U., & Rädiker, S. (2023). Using Software for mixed methods analysis. In R. Tierney, F. Rizvi, & K. Erkican (Eds.), International encyclopedia of education (4th ed., vol. 12, p. 500–512). Elsevier. https:// doi.org/10.1016/B978-0-12-818630-5.11049-8

Twelve different strategies for integrative mixed methods data analysis and its implementation with software packages are described. After an introduction of four types of integration (results-based, data-based, transformation-based and sequential integration) and the usage of joint displays, the focus is on the determinants of integration. For each type of integration, the use of software is discussed in detail. Onwuegbuzie, A. J., & Johnson, B. (Eds.). (2021). The Routledge reviewer’s guide to mixed methods analysis. Routledge.

This edited book contains numerous research examples on state-of-the-art mixed methods analysis, covering a broad thematic spectrum: (1) quantitative approaches to qualitative data; (2) qualitative approaches to quantitative data; (3) “inherently” mixed analysis approaches; (4) use of software for mixed analysis. Kuckartz, U., & Rädiker, S. (2019). Analyzing qualitative data with MAXQDA: Text, audio, and video. Springer Nature Switzerland.

326

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

This textbook contains a basic introduction to the MAXQDA software. One chapter is dedicated to mixed methods data analysis.

REFERENCES Bazeley, P. (2012). Integrative analysis strategies for mixed data sources. American Behavioral Scientist, 56(6), 814–828. https://doi.org/10.1177/0002764 211426330 Bazeley, P. (2018). Integrating analyses for mixed methods research. Sage. Boyatzis, R. E. (1998). Transforming qualitative information: Thematic analysis and code development. Sage. Cox, K., Lambert, R., & Hitchcock, J. H. (2021). Multiple linear regression analysis with qualitative data that have been quantitized. In A. J. Onwuegbuzie & R. B. Johnson (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 77–87). Routledge. Creswell, J. W., Klassen, A. C., Plano Clark, V. L., & Smith, K. C. (2011). Best practices for mixed methods research in the health sciences. https://obssr. od.nih.gov/sites/obssr/files/Best_Practices_for_ Mixed_Methods_Research.pdf Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Dickinson, W. B. (2021). Correspondence analysis of qualitative data. In A. J. Onwuegbuzie & R. B. Johnson, The Routledge reviewer’s guide to mixed methods analysis (pp. 37–44). Routledge. https:// doi.org/10.4324/9780203729434-3 Fetters, M. D., & Guetterman, T. C. (2021). Development of a joint display as a mixed analysis. In A. J. Onwuegbuzie & R. B. Johnson (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 259–275). Routledge. Guba, E., & Lincoln, Y. S. (1985). Naturalistic inquiry. Sage. Guest, G., MacQueen, K. M., & Namey, E. E. (2012). Applied thematic analysis. Sage. Guetterman, T. C., Creswell, J. W., & Kuckartz, U. (2015a). Using joint displays and MAXQDA software to represent the results of mixed methods research. In M. T. McCrudden, G. J. Schraw, & C. W. Buckendahl (Eds.), Use of visual displays in research and testing: Coding, interpreting, and reporting data (pp. 145–176). Information Age Publishing. Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015b). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. The Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm.1865

Haynes-Brown, T., & Fetters, M. D. (2018). The analytic process of developing a joint display in a mixed methods study: An example from the use of Information and Communication Technology (ICT) in secondary schools in Jamaica project. MMIRA International Conference, 22 August, Vienna. Hitchcock, J. H., Lambert, R., & Scott Holcomb, T. (2021). Hierarchical linear modeling with qualitative data that have been quantitized. In A. J. Onwuegbuzie & R. B. Johnson, The Routledge reviewer’s guide to mixed methods analysis (pp. 99–108). Routledge. https://doi.org/10.4324/9780 203729434-9 Hitchcock, J. H., & Onwuegbuzie, A. J. (2022). The Routledge handbook for advancing integration in mixed methods research. Routledge. www.taylorfrancis.com/books/9780429432828 Kelle, U. (2007). The development of categories: Different approaches in grounded theory. In A. Bryant & K. Charmaz (Eds.), The SAGE handbook of grounded theory (pp. 191–213). Sage. Kuckartz, U. (2014). Qualitative text analysis: A guide to methods, practice & using software. Sage. Kuckartz, U. (2017). Datenanalyse in der MixedMethods-Forschung: Strategien der Integration von qualitativen und quantitativen Daten und Ergebnissen. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie, 69(S2), 157–183. https:// doi.org/10.1007/S11577-017-0456-Z Kuckartz, U., & Rädiker, S. (2019). Analyzing qualitative data with MAXQDA: Text, audio, and video. Springer. https://doi.org/10.1007/978-3-030-15671-8 Kuckartz, U., & Rädiker, S. (2021). Using MAXQDA for mixed methods research. In R. B. Johnson & A. J. Onwuegbuzie (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 305–318). Routledge. https://doi.org/10.4324/9780203729 434-26 Kuckartz, U., & Rädiker, S. (2022). Using MAXQDA for integration in mixed methods. In J. H. Hitchcock & A. J. Onwuegbuzie, The Routledge handbook for advancing integration in mixed methods research (pp. 540–562). Routledge. https://doi. org/10.4324/9780429432828 Kuckartz, U., & Rädiker, S. (2023). Using Software for mixed methods analysis. In R. Tierney, F. Rizvi, & K. Erkican (Eds.), International encyclopedia of education (4th ed., vol. 12, p. 500–512). Elsevier. https:// doi.org/10.1016/B978-0-12-818630-5.11049-8 Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage. Onwuegbuzie, A. J., & Johnson, B. (Eds.). (2021). The Routledge reviewer’s guide to mixed methods analysis. Routledge. Onwuegbuzie, A. J., & Leech, N. L. (2019). On qualitizing. International Journal of Multiple Research Approaches, 11(2), 98–131. https://doi.org/ 10.29034/ijmra.v11n2editorial2

USING SOFTWARE FOR INNOVATIVE INTEGRATION IN MIXED METHODS RESEARCH

Onwuegbuzie, A. J., & Leech, N. L. (2021). Qualitizing data. In R. B. Johnson & A. J. Onwuegbuzie (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 141–150). Routledge. Péladeau, N. (2021). Cluster analysis for mixed methods research. In A. J. Onwuegbuzie & R. B. Johnson, The Routledge reviewer’s guide to mixed methods analysis (pp. 57–68). Routledge. https:// doi.org/10.4324/9780203729434-5 Rädiker, S. (2018). Interactive joint displays in MAXQDA for mixed methods data analysis. MMIRA International Conference, 22 August, Vienna. Sandelowski, M., Voils, C., & Knafl, G. (2009). On quantitizing. Journal of Mixed Methods Research, 3(3), 208–222. https://doi.org/10.1177/155868 9809334210 Schoonenboom, J., & Johnson, R. B. (2017). How to construct a mixed methods research design. KZfSS Kölner Zeitschrift Für Soziologie Und

327

Sozialpsychologie, 69(S2), 107–131. https://doi. org/10.1007/s11577-017-0454-1 Schreier, M. (2012). Qualitative content analysis in practice. Sage. Van Haneghan, J. (2021). Exploratory factor analysis of text. In A. J. Onwuegbuzie & R. B. Johnson (Eds.), The Routledge reviewer’s guide to mixed methods analysis (pp. 25–35). Routledge. van Velzen, J. H. (2018). Students’ general knowledge of the learning process: A mixed methods study illustrating integrated data collection and data consolidation. Journal of Mixed Methods Research, 12(2), 182–203. https://doi.org/10.1177/ 1558689816651792 Vogl, S. (2019). Integrating and consolidating data in mixed methods data analysis: Examples from focus group data with children. Journal of Mixed Methods Research, 13(4), 536–554. https://doi.org/ 10.1177/1558689818796364

23 Grounded Text Mining Approach: An Integration Strategy of Grounded Theory and Textual Data Mining Mitsuyuki Inaba and Hisako Kakai

In this chapter, our description and illustration of a grounded text mining approach (GTxA) offer an innovative, mixed methods analytical approach transcending data typically categorized otherwise as either qualitative or quantitative. This approach is a hybrid strategy integrating two methods (Bazeley, 2018)—namely, constructivist grounded theory (Charmaz, 2014) and textual data mining (hereafter text mining), for analyzing textual data. The GTxA improves the quality of researchers’ textual data analysis by combining their analytical and creative insights with a computer’s objectivity. We put forward three reasons why the GTxA may be of interest to the field of mixed methods research and how it can contribute to the current mixed methods literature by adding a new perspective and an analytical tool. The reasons are as follows. The first reason relates to data types used in mixed methods studies and has implications for how we conceptualize the methodology of mixed methods research. In general, researchers in the mixed methods field regard the use of both qualitative and quantitative data as necessary elements of mixed methods enquiry. Meanwhile, there has been a debate over the relevance of distinguishing between quantitative and qualitative data (e.g., Hammersley, 2018; Maxwell, 2019; Morgan, 2018; Sandelowski, 2018). In this debate, Sandelowski reminds us of the fluid nature of data

by referring to her previous research, indicating that “data are neither QL [qualitative] nor QN [quantitative], but rather[,] aspects of experiences or phenomena transformed into words, numbers, visual forms, and the like, each of which may, in turn, be transformed again into other forms” (Sandelowski, 2014, p. 5). Therefore, distinguishing between the qualitative and quantitative data types may be of scant significance because both data can be converted into the other type depending on the purpose of the study (e.g., Bazeley, 2018; Sandelowski, 2014). The second reason is that even one data type can be complex and profound. For example, textual data in the online ethnography of social networking services (SNS), confession statements in criminal justice, historical records, political documents and the like require a comprehensive data analysis procedure by way of an iterative examination of the same data (Inaba & Kakai, 2019; see also Chapter 10, this volume). We believe that by rendering the conceptualization and procedures of mixed methods more flexible and applicable to real-world situations, we can analyse such complex textual data effectively. The third reason stems from rapid advances in the support technologies and tools for textual data analysis in mixed methods (Bazeley, 2009b). Recent advances in computer-assisted

GROUNDED TEXT MINING APPROACH

qualitative data analysis software (CAQDAS) such as MAXQDA and NVivo include not only qualitative coding, but also complex textual data analysis features such as text visualization and automated theme extraction for use in mixed methods (Bazeley, 2009b, 2018; Guetterman et al., 2015). Additionally, text mining and related natural language processing (NLP) technologies are being actively discussed as tools that can accelerate the execution of mixed methods enquiry (Chang et al., 2021; Poth et al., 2021) and achieve its methodological extension (Creswell & Plano Clark, 2018; Fetters & Molina-Azorin, 2017). For further discussion of using MAXQDA software in mixed methods research, see Chapter 22 (this volume). The grounded text mining approach echoes this integrative thinking with complexity regarding mixed methods to respond to our dynamic realities proposed by Poth (2018). The approach attempts to move beyond the simple quantitative–qualitative dichotomy using an innovative approach that integrates grounded theory and text mining. These features of the GTxA mark “a more complexitysensitive approach to mixed-methods research” (Poth, 2018, xxviii), and it is a powerful tool for analyzing real-life data in natural settings. In the following sections, we first review Charmaz’s constructivist grounded theory and explain why we incorporate this version of grounded theory in the GTxA. Second, we shift our focus to a new integrated methodology called mixed methods grounded theory (MMGT) (Johnson et al., 2010) and a fully integrated MM-GT methodology (FIMM-GTM) (Creamer, 2022) to better explore the mixed methods origins of the GTxA. Third, we demonstrate the analytical procedure of the GTxA using focus group interview data obtained in our study investigating the challenges nursing researchers face in conducting mixed methods research to. Fourth, we discuss ethical issues involving the use of the GTxA. Finally, we discuss the implications and future directions of the GTxA, including its contributions to the mixed methods literature, its potential as an analytical approach to social and behavioural sciences and its limitations.

THE CONSTRUCTIVIST ORIGINS OF THE GROUNDED TEXT MINING APPROACH In this section, we introduce the origins of the GTxA as a strategy for integrating grounded theory and text mining. First, we briefly review grounded theory, with a particular emphasis on Charmaz’s constructivist version, followed by a

329

discussion of the rationale for using this constructivist version of grounded theory in the GTxA. Grounded theory originates from the collaborative research of two sociologists, Barney Glaser and Anselm Strauss, at the University of California, San Francisco, in the 1960s (e.g., Glaser & Strauss, 1967). The two researchers presented systematic and inductive data-collection guidelines for field research and data analysis the better to develop a middle-range theory that elucidates practical problems and basic social processes found in the phenomena under study (Charmaz, 2014; Glaser, 1978; Glaser & Strauss, 1967; Strauss & Corbin, 1998). Charmaz’s constructivist approach to grounded theory emerged from her epistemological stance that challenged the positivistic assumptions embedded in the classic approach to grounded theory developed by Glaser and Strauss (1967), as well as the Glaserian and Straussian grounded theory approaches that followed. Charmaz (2000, 2014) referred to these approaches to the grounded theory as the “objectivist grounded theory” and differentiated it from her “constructivist grounded theory”. According to Charmaz, objectivist grounded theory assumes the existence of a single social reality, which is “discovered” in data, regardless of who collects or analyses the data. This assumed neutrality between the researcher and the researched is believed to provide an objective description of causes, conditions, consequences and predictions of an event or action, or the resolution of a major problem. Contrary to this stance, Charmaz’s constructivist grounded theory assumes that data are co-constructed by the researcher and the researched through their interactions and analysed through the eyes of the researcher. As a result, a theory resulting from the researcher’s investigation is considered an interpretive rendering of the world. Charmaz’s constructivist approach shares several attributes with objectivist grounded theory, including simultaneous data collection and analysis, the construction of analytic codes and categories from data, the use of constant comparative methods, theory-building orientation, a focus on action and process, the use of memo writing to elaborate categories and define relationships between the categories, and the use of theoretical sampling. However, the epistemic differences in the objectivist and constructivist versions of grounded theory also lead to their differences in research practice, ranging from the initial research objectives to the nature of the given “theory” to be developed (Charmaz, 2000). Of particular interest to us in terms of the distinctions between objectivist and constructivist versions of grounded theory are the coding strategies

330

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

of the constructivist version due to their simplicity and flexibility. Strauss and Corbin (1998), whose approach Charmaz regarded as exercises in objectivist grounded theory, used complex coding strategies involving multiple tools (e.g., axial coding, paradigm and condition/consequences matrix). They proposed the use of these tools to enable researchers to examine data from different aspects to avoid superficial analysis. However, the coding strategies of constructivist grounded theory are significantly simpler and more flexible than those of objectivist grounded theory approaches. They require researchers to immerse themselves in the world of research participants’ experience. Charmaz believed that directing the researcher’s attention to cumbersome coding strategies simplistically aiming for more objectivity would hinder the researcher’s focus on actually listening to the participants’ voices. As a result, Charmaz’s approach mainly uses the following two very simple coding strategies: initial coding and focused coding, and details of these coding strategies will be discussed later. In summary, Charmaz’s coding strategies were the most influential in shaping GTxA. Charmaz’s simple coding strategies and computer-based coding complement each other because the former allows researchers to use their insights fully by immersing themselves in research participants’ experiences, whereas the latter offers objective views of the data through visualization during analysis.

THE MIXED METHODS ORIGINS OF THE GROUNDED TEXT MINING APPROACH Thus far, we have explained the characteristics of Charmaz’s constructivist grounded theory, providing a methodological basis for the GTxA. In this section, we locate the GTxA in the mixed methods literature that integrates mixed methods and grounded theory. Integration of grounded theory and mixed methods reflects the recent expansion in the notion of integration and its deployment in mixed methods research (Fetters & Molina-Azorin, 2017; Greene, 2015). Researchers intersect (Plano Clark & Ivankova, 2016) or scaffold (Fetters, 2020) multiple methodologies to develop an innovative approach to mixed methods research that best fits their research purposes. Integrating grounded theory and mixed methods is also endorsed by Charmaz and colleagues with the following statement: “Grounded theory and mixed methods may enjoy a symbiotic relationship. Grounded theory can contribute to mixed methods methodology and practice

(e.g., within a two-phase approach); mixed methods methodology may also contribute to a grounded theory study” (Charmaz et al., 2018, p. 433). Johnson et  al. (2010) first proposed a mixed methods version of grounded theory (MM-GT). They claimed that grounded theory is sufficiently compatible with any of the three forms of mixed methods enquiry, including qualitative dominance, equal status and quantitative dominance. However, Johnson et al. argued that, among these three forms, MM-GT works best in equal-status mixed methods by enabling researchers to engage in “connecting theory generation with theory testing, linking theory and practice, and linking general/nomological description/explanation with idiographic understandings of the human world” (p. 65). The idea of integrating mixed methods and grounded theory has been further expanded by the scholarly works of other mixed methods researchers. For example, Creamer (2018) proposed extending MM-GT by introducing the concept of fully integrated mixed methods. She argues that, in MM-GT methodology (MM-GTM), the integration of quantitative and qualitative procedures ordinarily only occurs at the final stage of inferencing but that it should, instead, occur at multiple stages of the research process. Creamer calls this extended version a “fully integrated mixed method grounded theory methodology” (FIMMGTM), which “uses grounded theory procedures and strategies to integrate different sources of data throughout the research process, including during data analyses, to produce and sometimes to test a theoretical framework” (Creamer, 2022, p. 8). Guetterman et  al. (2019) systematically reviewed the MM-GT literature and found that the approach was widely used in various disciplines. Among 61 MM-GT empirical articles identified by the researchers, two articles used Charmaz’s constructivist version of grounded theory in MM-GT: Pieterse et al. (2016) and Sherbino et al. (2014). Of those two articles, Pieterse et al. (2016) articulated their rationale for using a constructivist version of grounded theory and explained its analytical procedures. They emphasized the characteristics of Charmaz’s version as acknowledging that “both the participants’ understandings of their reality and the researchers’ understandings of the data are interpretive; that is, they reflect the participants and researchers’ understandings, rather than objective truths” (p. 33). Furthermore, although Guetterman et al. (2019) found that a limited amount of MM-GT research presented detailed methodological discussions on mixed methods, grounded theory and the integration of these two methodologies, leading to theoretical development, they recognized the benefits of integrating the two methodologies. For a further

GROUNDED TEXT MINING APPROACH

discussion of best practices for MM-GT for design and implementation. Based on their review of the MM-GT literature, the researchers highlighted the benefits of using MM-GT for both grounded theorists and mixed methods researchers. These benefits demonstrate the complementary nature of mixed methods and grounded theory, echoing the statement of Charmaz et al. (2017) introduced at the beginning of this section. In this section, we reviewed the mixed methods origins of the GTxA. The literature supported the value of integrating grounded theory and mixed methods approaches. Furthermore, it was also indicated that Charmaz’s version of grounded theory might serve well when integrated with an objective quantitative approach in mixed methods (Pieterse et al., 2016). In conclusion, we propose the GTxA as an innovative development of MM-GT that can expand the possibility of mixed methods analysis.

OVERVIEW OF THE GROUNDED TEXT MINING APPROACH The GTxA is a hybrid strategy for analyzing textual data both qualitatively and quantitatively for theory building. It combines both text mining techniques and the constructivist version of the grounded theory analysis. This strategy enables researchers to perform comprehensive data-driven and in-depth analysis of textual data using computer technology.

Text mining techniques aid researchers in analyzing textual data by automating data construction and analysis, providing an opportunity to determine whether the data support the interpretation in question. If a discrepancy emerges between the text mining results and the researcher’s interpretation of the data, the researcher can return to the raw data and carefully refine the codes. This iterative process of data analysis in the GTxA helps researchers gain an in-depth understanding of textual data while minimizing biases distorting the interpretation. During the analysis process, using Charmaz’s simple coding system, researchers can be open-minded, insightful, imaginative and creative in their thinking, while focusing on listening to research participants’ voices. Text mining analysis works simultaneously with researchers’ grounded theory analysis, providing them with different perspectives on the phenomenon that would not have been visible otherwise—that is, by allowing the collaboration between humans and computers, the GTxA attempts to achieve what the Glaserian and Straussian versions of grounded theory are trying to achieve using complex coding strategies.

THE ITERATIVE PROCESS OF THE GROUNDED TEXT MINING APPROACH Figure 23.1 illustrates the framework and iterative analysis process of the GTxA. In Step 1, a researcher

Step 1: Obtaining an overview of text data

Step 4: Meta-Inference

Step 2: Qualita�ve Data Analysis with GT

Compare & modify

Step 3: Quan�ta�ve Visualiza�on & Analysis

Conversion (Quan��zing with auto-coding)

Figure 23.1  Framework and iterative process of GTxA Source: Author created

331

332

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

obtains an overview of the textual data. It is helpful in this step to use text mining tools (i.e., software) to visualize the frequency of word occurrences and the relationships of words that co-occur. In Step 2, the researcher conducts a qualitative data analysis using the simple coding systems of constructivist grounded theory, including initial coding and focused coding. As a result, a set of initial codes (i.e., in vivo codes with word-byword coding) and focused codes are developed. In the first part of Step 3, auto-coding rules for using text mining tools are defined based on a combination of in vivo codes and focused codes generated in the previous step. Then, textual data are quantitized using focused codes by applying the rules to data. This quantitizing (Sandelowski et  al., 2009) enables the researcher to perform a statistical analysis of differences in the frequency of focused codes according to variables such as speakers or groups. In Step 4, the results of the focused coding that the researcher performed in Step 2 and the automatically quantified results using the text mining software are examined. Ambiguous aspects of the coding that the researcher performed in Step 2 may emerge in this process. It is also possible for automatic computer processing to produce inappropriate results. When encountering such cases, the researcher returns to the previous analytical steps and refines their codes. The auto-coding rules are then revised until the quantified results of the automatic processing of the computer become consistent with the researcher’s manual coding. This iterative analytical process may provide the researcher with deeper insights into what the most appropriate interpretations of the textual data are, based on the integrated results of the grounded theory and text mining analyses. Therefore, this process enables the researcher to gain a better understanding of the phenomenon than would be obtained using solely either a grounded theory or a text mining approach. In summary, the GTxA helps researchers compare analysis results of textual data both qualitatively and quantitatively. It also assists them in drawing meta-inferences developed by integrating the results of the two different analytical approaches.

The Epistemological Stance Taken When Using the Grounded Text Mining Approach Because the GTxA involves textual data transformation, it reflects one of the characteristics of the

“crossover mixed analysis” (Hitchcock & Onwuegbuzie, 2020; Onwuegbuzie & Combs, 2010; Onwuegbuzie & Hitchcock, 2015)—that is, the use of methods crossing over the paradigms of qualitative and quantitative approaches. The GTxA is an example of a “qualitative– dominant crossover analysis” or “qualitatively driven mixed methods” (Hesse-Biber, 2010; Hesse-Biber et  al., 2015) because its use of grounded theory coding plays a critical role in the entire analytical process, while integrating the results of qualitative data analyses and the frequency of themes with computer-based autocoding to gain an in-depth understanding of the studied phenomenon. Although the GTxA uses the coding system of constructivist grounded theory, epistemologically, it lies in the middle of objectivist and constructivist grounded theories. In other words, “a researcher who uses GTxA believes that theories can be either plausible causal explanations [i.e., an objectivist grounded theory] or interpretations of the meanings and actions of the study participants [i.e., a constructivist grounded theory]” (Inaba & Kakai, 2019, p. 335, square brackets added). Therefore, one’s research questions and goals determine the type of theory one develops.

Key Terms and Descriptions of a Case Study First, we present GTxA terms and their definitions. Then, we describe the case study from which textual data that we use in this section originated. Here, we provide a real-data illustration of a GTxA analysis using data collected from focus group interviews with nursing graduate students and professors in Japan. Table 23.1 shows key GTxA terms and their definitions. The reader should be familiar with these terms before reading a case study of the GTxA.

A CASE STUDY OF THE GTXA: THE MIXED METHODS CHALLENGES FACED BY NURSING RESEARCHERS To illustrate the GTxA procedure, the following section presents a case study of the analysis of textual data collected from focus group interviews with nursing graduate students and professors in Japan. These focus group interviews were conducted as part of a government-funded project aiming to reveal obstacles that nursing

GROUNDED TEXT MINING APPROACH

333

Table 23.1  Key terminologies and definitions Computer-assisted CAQDAS is “well understood across disciplines as broadly referring to software designed to qualitative data analysis assist the analysis of qualitative data” (Silver & Lewins, 2014). Recent CAQDAS such as software (CAQDAS) MAXQDA and NVivo facilitates not only the coding of qualitative data, but also complex textual data analysis features such as text visualization and thematic analysis for mixed methods research (Bazeley, 2018, 2019b; Guetterman et al., 2015). Natural language NLP is one of the fundamental technologies of artificial intelligence. “Natural language” is processing (NLP) the language that we use for everyday reading, writing and speaking. In recent years, various tools have become available to analyse vast amounts of textual data in natural language, both in research and business. NLP has also gained particular attention for the accelerated execution of mixed methods research (Chang et al., 2021). KH Coder

Auto-coding

KWIC (Key words in context) concordance Initial coding

Focused coding

KH Coder is open-source software for text mining of multilingual resources. Monkin is a series of add-on software that provides functions to process synonyms and negative expressions to KH Coder. KH Coder has been used recently in the context of mixed methods research (Hoshino & Suwa, 2019; Yamauchi et al., 2021). Auto-coding is the computerized automatic coding of textual data according to humandefined rules or patterns. For example, a user creates rules such as assigning the code “mentoring” to documents that contain “mentor”, “guide” or “guidance”. It is also possible to define more complex rules that combine multiple conditions using AND, OR, and NOT logical operators. In recent years, such functionality has been provided using text mining tools such as KH Coder (Higuchi, 2016) as well as using CAQDAS. KWIC concordance is a feature that displays “the occurrences of a chosen word with its surrounding context” (Biber et al., 1998, p. 26). Similar functions are available in many text mining tools and CAQDAS. The researcher starts with initial coding when analyzing textual data in constructivist grounded theory. By carefully reading textual data, the researcher remains open to any ideas that may lead to theoretical development. Initial coding includes, but is not limited to, word-by-word coding and line-by-line coding. With word-by-word coding, the researcher codes symbolic words. With line-by-line coding, the researcher codes each line of data by assessing what is happening and what kind of theoretical implication the data suggest (Charmaz, 2014). Focused coding is the second stage in data analysis of constructivist grounded theory. It means using the most significant and frequent initial codes to sort through a large amount of data. This coding involves determining which initial codes make the most analytical sense (Charmaz, 2014).

Source: Author created.

researchers may face when planning and conducting mixed methods studies to develop guidelines and e-learning materials of mixed methods for nursing research.1 We used data obtained from two focus groups—namely, a group of graduate students with no experience in conducting mixed methods studies and a group of professors with prior mixed methods experience, drawn from the eight focus groups and four individuals we interviewed. We demonstrate how the GTxA analyses can help deepen our understanding of textual data by presenting the similarities and differences across the trajectories of the participants in the two groups. During each focus group interview, we asked several questions regarding the participants’ perceptions of or experiences with mixed methods. In this chapter, we will use part of the data

(i.e., perceived challenges when planning and conducting mixed methods in nursing research) to describe the GTxA procedure. The interviews were conducted and transcribed in Japanese.2 Each focus group interview lasted approximately two hours. The participants’ responses in the two focus groups consisted of 512 sentences and 139 paragraphs of text.

FOUR PROCEDURES OF THE GROUNDED TEXT MINING APPROACH In this section, we will explain in detail the iterative process of the GTxA using the aforementioned focus group interview data collected from nursing researchers. There are four steps in this

334

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

process: Step 1, capturing an overview of the textual data; Step 2, conducting a qualitative data analysis using grounded theory; Step 3, quantitatively visualizing and analyzing textual data; and Step 4, drawing meta-inferences.

Step 1: Capturing an Overview of the Textual Data Step 1 of the GTxA obtains an overview of the textual data. To proceed with this task efficiently, it is practical to use text mining software, which has been advancing rapidly in recent years. In this case study, we used KH Coder, which is free text mining software developed in Japan, to analyse the verbatim transcripts of the focus group interviews. The software is capable of analyzing data in multiple languages. Its homepage is available in both Japanese3 and English.4 We also used Monkin,5 which is an add-in that extends the visualization and synonym processing of KH Coder.

Figure 23.2 presents an example of analyzing the responses in the focus groups with the students (G1) and professors (G2) using the correspondence analysis with KH Coder and Monkin software.6 The respondents of the two focus groups commonly mentioned words that appeared around the centre point of the graph. The words that appear from the centre point towards the lower left are words characteristic of the students’ focus group, whereas those towards the upper right are those of the professors’ focus group. The words “learn”, “research question”, “master” and “research methods” appear in the lowerleft corner of Figure 23.2. These words relate to the challenges that students face in using mixed methods. We used the key words in context (KWIC) concordance of KH Coder to understand the specific contexts in which these words were used. For example, the context of the phrase “research question” was extracted, as shown in Figure 23.3. The following is one of the extracted responses regarding the “research question”:

Figure 23.2  Correspondence analysis of challenges in practising mixed methods research Source: Author created.

GROUNDED TEXT MINING APPROACH

335

Figure 23.3  Context of the phrase “research question” Source: Author created. As for the research question, in my opinion, you have to have an idea of what you want to explore, and then you have to figure out which method is appropriate. This is not only difficult in mixed methods research, but I always feel that it is difficult in all research processes. In other words, I don’t think that mixed methods research makes research questions more difficult. You don’t decide to use mixed methods before developing research questions; the research questions come first, and then the research methods are selected accordingly. (Speaker: graduate student S2)

According to this statement, student S2 is still learning about research methods. Therefore, this student describes the challenge as the difficulty in formulating a research question and choosing the appropriate research method for it. This challenge is not specific to mixed methods; rather, it is applicable to research methods in general. Conversely, words such as “timing”, “trouble” and “drawing” appear in the outermost part of G2 (professors) in the upper right corner of Figure 23.2. These words and phrases relate to challenges that professors perceive in terms of mixed methods research. As an example, some contexts containing the word “timing” are shown in Figure 23.4. The following is one of the extracted responses regarding “timing”: In qualitative research, however, it is difficult to know how well the students can find the right

timing [for data collection]. In addition, they may not be able to go to the field every day, so there is a possibility that they may miss the timing because of that … (Speaker: Professor P1)

Here, Professor P1 mentions the difficulty in collecting data at the right timing, especially through qualitative research. This is because students engaged in the study did not go to the clinical site every day. As a result, they missed the right time to collect data from patients. This is a practical issue in nursing research. As previously indicated, the first step of the GTxA is to use text mining software to analyse the entire text and then use the KWIC concordance to identify contexts that contain feature words. By doing this, an analyst can obtain a quick overview of the entire text data.

Step 2: Conducting a Qualitative Data Analysis using Grounded Theory Step 2 of the GTxA adopts the constructivist version of the grounded theory method proposed by Charmaz (2014), which emphasizes the researcher’s insights and interaction with the textual data. In this step, the researcher reads the entire text data and develops codes and categories to derive a theory using the coding system of the constructivist grounded theory.

336

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 23.4  The context for the keyword “timing” Source: Author created.

In the process of implementing the constructivist version of the grounded theory, the researcher conducts initial coding (word-by-word coding and line-by-line coding) while reading textual data. In the following focused coding, the researcher develops categories of social psychological events that they believe should be the object of focus from the initial coding results. Various CAQDAS packages (e.g., MAXQDA, NVivo, and ATLAS.Ti) are available for conducting grounded theory. Figure 23.5 shows a screenshot of the coding process with grounded theory using MAXQDA 2020 (VERBI Software, 2019). Table 23.2 presents the construction process of codes from initial coding to focused coding. The researcher first obtains characteristic words from each line in the transcript via word-by-word coding. Next, line-by-line coding is performed to summarize the codes extracted in the previous stage. Then, in focused coding, the researcher develops a group of codes or categories for in-depth analysis. Following the analysis using constructivist grounded theory, 16 focused codes were generated. These codes and the original text were then reexamined, yielding three hierarchical code groups regarding the mixed methods hurdles encountered in nursing research (Figure 23.6). In other words, the hurdles of mixed methods research and those of nursing research are embedded in the challenges of research methods in general. Code group I represents challenges in learning the general research methods, and they are not limited to mixed methods research. This code group includes “RQs & methods”, “mentoring”,

“lack of knowledge” and “IRB”, which are necessary for learning any research method. Code group II includes challenges directly related to mixed methods learning and practice. For example, “integration”, “design guide” or “visualization guide” are particular to mixed methods research. Code group III includes challenges, especially those related to research in the nursing field. For example, a participant stated that it was difficult to collect data with appropriate “timing” in nursing. This issue is especially relevant to conducting a longitudinal mixed methods study in nursing (Plano Clark et al., 2015). Thus, constructivist grounded theory analysis enables researchers to understand a group of challenges directly or indirectly related to mixed methods research in nursing research, ensuring that there is no discrepancy between researchers’ interpretation of data and the original textual data.

Step 3: Quantitatively Visualizing and Analyzing Textual Data Step 3 of the GTxA verifies the relevance of codes developed using grounded theory deployed in the previous step by comparing them with codes generated using computers. In the initial phase of Step 3, auto-coding is performed using text mining software to visualize the features of textual data qualitatively and quantitatively. Rules for auto-coding are defined as a combination of the results of the word-by-word coding

337

GROUNDED TEXT MINING APPROACH

Figure 23.5  A screenshot of qualitative data analysis with MAXQDA Source: Author created.

Table 23.2  Examples of focused coding (mentoring) No

FG

Speaker Content

Initial coding: word-by-word coding

Initial coding: line-by-line coding

Focused coding

69

G1

IA

Thus, what do you think is necessary to overcome the challenges that you have just mentioned? S4, what do you think is necessary?

70

G1

S4

I think it’s important to have someone to guide you to make sure that your method is correct.

Guide: make sure that your method is correct

It is important to have a mentor on the method.

Mentoring

71

G1

S4

First, I do not have any mentors who specialize in mixed-methods research around me. Hence, I do not have many opportunities to receive direct guidance. If that is the case, I think the best thing to do would be to have a seminar or something where you can get guidance.

Mentors: direct guidance, get guidance

I want a mentor who can give me direct guidance on the method.

Mentoring

Source: Author created.

338

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 23.6  Three hierarchical code groups of hurdles in mixed methods nursing research Source: Author created.

and focused coding in Step 2. For example, by obtaining the results of the word-by-word coding and focused coding in Table 23.1, researchers define auto-coding rules for KH Coder, as shown in Figure 23.7. Here, those words or phrases extracted through word-by-word coding in Step 2 are listed using the ‘OR’ condition. These words and phrases are bonded with the label “mentoring”, which has an asterisk (*) on its left, as shown in Figure 23.7. This label indicates researchers’ focused codes.

Figure 23.8 is based on a cross-tabulation table of codes and the frequency of their occurrence, which was obtained using the aforementioned KH Coder’s auto-coding rules. Figure 23.8 presents the chi-square test results on the left side of each code. The residual analysis results are shown on the right side of each code. In this analysis, the residuals of each row were standardized. Triangles pointing upwards are significantly larger than those pointing downwards. The “Pearson rsd” in Figure 23.8 indicates the

＊ Mentoring mentors OR guide OR guidance OR ‘get advice’ OR ‘senior colleague’ OR ‘make sure that your method is correct’

Figure 23.7  An example of auto-coding rules for KH Coder Source: Author created.

GROUNDED TEXT MINING APPROACH

339

Figure 23.8  Results of auto-coding using KH Coder Source: Author created

standardized residuals calculated for each code. A darker colour in a focus group indicates that it contains more codes than the other focus group. “Per cent” indicates the percentage of codes appearing in each focus group. The larger the plot size, the higher the percentage of code occurrences in that group. Next, we present the specific results, as shown in Figure 23.8. The frequency of the focused code “RQs & methods” is significantly higher in the students’ focus group (G1) than that in the professors’ focus group (G2), both using the chi-square test and residual analysis. Figure 23.8 also demonstrates that G1 is significantly higher than G2 for the “mentoring” and “lack of knowledge” categories using residual analysis. Specifically, the results indicate that the students strongly perceive these three hurdles during mixed methods implementation. Furthermore, Figure 23.8 suggests that the professors’ focus group (G2) has a higher need

for the “design guide”, “visualization guide” and the “writing guide”, and has a significantly higher degree of perceiving the “timing” of data collection as a hurdle. During the GTxA, the researcher may find a discrepancy between the qualitative data analysis results in Step 2 and auto-coding results in Step 3. In these cases, the researcher should go back to Step 2 and recode the textual data using a grounded theory approach. Based on the results, the researcher revises the coding rules for autocoding in Step 3. The GTxA aims to integrate findings derived from grounded theory analysis using researchers’ insights and the results of the objective quantification of textual data using computer-based auto-coding. The researcher continues this iterative analytical process until there is no discrepancy remaining between the findings of the two different approaches.

340

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Step 4: Drawing Meta-inferences In Step 4 of the GTxA, the researcher reviews similarities and differences in the results between the two groups to obtain meta-inference to the results of the qualitative data analysis in Step 2 and the quantification of the same data in Step 3. Figure 23.9 is a visual representation of the meta-inferences in the GTxA. Figure 23.9 is the similar to Figure 23.8. The only difference between Figure 23.8 and Figure 23.9 is that Figure 23.9 has three different code categories distinguished by rounded rectangles drawn with dotted lines. Code group I represents hurdles in research methods in general, code group II those found in mixed methods research, and code group III those found in nursing research. From Figure 23.9, the focused codes of code groups I and II seem to be related to the experience

of the students’ group (G1). Upward triangles indicate large residuals; this means that the three codes of “RQs and Methods”, “Mentoring” and “Lack of knowledge” emerged from G1’s discussion with relatively high frequency. Simultaneously, students’ hurdles seem directly related to mixed methods practice in their graduate courses, including codes such as “Confusing terminology” and “Short training period”. This implies that there is divergence in the hurdles the students perceive in their research practices, ranging from challenges related to research methods in general to those peculiar to mixed methods research. Conversely, there was scant mention of codes related to code group I in the professors’ group (G2). In this group, the participants’ utterances appeared to converge on one important topic, which is a guide for teaching mixed methods research. This observation may be supported

Figure 23.9  Integrated results of grounded theory analysis and auto-coding Source: Author created.

GROUNDED TEXT MINING APPROACH

by the three codes of “Writing guide”, “Design guide” and “Visualization guide” appearing with a significantly higher frequency in G2 than in G1. These codes seem to reflect the professors’ strong desire to teach students how to conduct mixed methods enquiry concretely and appropriately. Furthermore, relatively large residuals on codes specific to mixed methods practice in nursing were found in G2. For example, the code “timing” refers to the appropriate timing for data collection in a longitudinal mixed methods study in the nursing field. In other words, professors’ hurdles may be directly related to teaching and practising mixed methods research in nursing.

SUMMARY In summary, meta-inferences drawn from this GTxA study reveal major hurdles in mixed methods research rigorously, as shown in Figure 23.9. We regard Figure 23.9 as a type of “joint display” that illustrated the integrated results of both “qualitative” and “quantitative” data analyses. Furthermore, the joint display not only informs us of major hurdles in mixed methods research perceived by the participants but also how these hurdles are related to knowledge and skills in general research methods as well as those practical and day-to-day issues surrounding nursing research practice. Various hurdles in mixed methods research exist in a continuum, ranging from those typically encountered by novice researchers to those encountered by experienced researchers. Thus, to provide effective support for nursing researchers interested in conducting mixed methods enquiry, it is insufficient to offer textbook knowledge regarding mixed methods research. Instead, it may be necessary to offer basic principles of research methods for novice researchers, and offer more practical and concrete knowledge applicable to case-by-case situations peculiar to nursing practice for more experienced researchers. Providing professors with teaching guides that instruct students on how to design and conduct mixed methods studies and methods to create joint displays within the context of nursing research may also be required. To illustrate the GTxA procedure, we only analysed the interview data of the two focus groups in this chapter. However, in general, a researcher using the GTxA can continue to collect data using a theoretical sampling strategy and compare and contrast the collected data until no new category is found (i.e., data saturation). When a researcher reaches

341

the point of data saturation, the patterns discovered in the data may constitute a grounded theory.

KEY ETHICAL ISSUES IN THE GROUNDED TEXT MINING APPROACH Currently, various studies and discussions on the ethical issues of mixed methods research exist (Cain et  al., 2019; Preissle et  al., 2015; Stadnick et  al., 2021). The literature also addresses issues arising from mixed methods research’s unique characteristic of integrating multiple analyses. There are also challenges as mixed methods research strives for social justice (Mertens, 2007). The GTxA, as a type of mixed methods analysis, shares some similarities to the challenges described in the literature. In addition to these challenges, there are three main ethical issues unique to the GTxA that researchers should consider when using the approach. First, there are issues related to the rights of the authors of the textual data to be analysed using the GTxA. For example, it is easy to analyse various text data circulating on SNS using text mining techniques. As a result, the researcher may collect such textual data from the internet without obtaining informed consent (IC) from the authors. However, in some cases, it is necessary to obtain IC from an author of textual data on the internet, or some electronic bulletin boards have a policy that the operator of the site owns the copyright rather than the data creator. Therefore, researchers should be cautious about issues of rights when they use the GTxA to analyse textual data available on the internet. Note that the case study presented in this study uses textual data of a research project that obtained the approval of the authors’ institutional review board, and we obtained IC from the study participants at the time of data collection. Second, there are issues related to the use of text mining software for data analysis by the GTxA. By combining text mining techniques with grounded theory, the GTxA seeks to ascertain whether researchers have inappropriately interpreted or distorted the intention of the original textual data in their grounded theory analysis. Additionally, because the GTxA uses text mining software, another problem is associated with software-based data analysis. Certain words and phrases are automatically selected or eliminated in analytical processes because analysis algorithms and dictionaries of text mining tools contain the design ideas of software developers. This affects the results of the analysis conducted using such tools. Therefore, during

342

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

the GTxA, it is ideal to use multiple text mining tools for triangulation. Moreover, researchers must be aware of the limitations that analytical tools may introduce into their research. Although these tools enable researchers to accomplish certain tasks, they may also impose limits on what they can do (Kakai, 2022; Wertsch, 1998). Finally, the third issue concerns the reporting phase. One of the characteristics of the GTxA is that the combination of grounded theory and text mining ensures the verifiability of the integrated results. Therefore, researchers need to disseminate details of how they integrate the results of the two analysis approaches. In other words, researchers should present as much detailed information about the decision-making process as possible.

IMPLICATIONS AND FUTURE DIRECTIONS This chapter illustrated how the GTxA can rigorously analyse textual data collected in natural settings. The GTxA contributes to the mixed methods literature by ensuring rigorous qualitative analysis results and expanding the possibility of qualitative data analysis to go beyond just theme analysis (Bazeley, 2009a). Although the dominant discourse in the mixed methods literature requires the collection of two different data types (i.e., quantitative and qualitative data), this chapter has demonstrated that applying two different analytical approaches to a single dataset can also draw meta-inferences or synergistic knowledge by integrating the results of multiple analysis approaches. Recently, in the Journal of Mixed Methods Research, an example of an empirical study that involves only one strand of data collection (i.e., focus group interview data) but uses multiple strategies to analyse data has been introduced as an example of mixed analysis (Vogl, 2019). The author argues that the “practical problems of data integration and consolidation are significantly similar in mixed methods analysis based on two (or more) strands of data collection and mixed analysis based on one data collection” (Vogl, 2019, p. 551). Echoing the view of Vogl about the use of multiple analysis approaches for one type of data, the GTxA is a mixed methods analysis even though it involves only one strand of data collection. Furthermore, in the GTxA, data integration occurs at the multiple levels of philosophy, methodology and methods for developing synergistic knowledge (Fetters & Molina-Azorin, 2017; Greene, 2015) as we have demonstrated in this chapter.

Fetters and Freshwater (2015) expressed the synergistic nature of mixed methods research as “1 + 1 = 3” (p. 116), indicating that the integration of qualitative and quantitative studies yields more than its separate elements. Although this equation is significantly simple, it accurately describes the most pertinent characteristics of mixed methods research, that is, mixed methods’ ability to provide synergistic findings to better comprehend the phenomenon under investigation. Meanwhile, Bazeley (2018) stated that “[n]ot all methods can be neatly categorized as qualitative or quantitative. Inherently mixed or hybrid methodologies that merge elements of both qualitative and quantitative approaches, which are neither one nor the other, have been and are continuing to be developed” (p. 260). We believe what defines a data type as quantitative or qualitative is the data analysis approach. Texts can be quantitative or qualitative depending on how we deal with them in the analysis; this demands the use of a hybrid iterative strategy, such as the GTxA, for data analysis. As a result, the meaning of the equation representing mixed methods synergy, “1 + 1 = 3” (Fetters & Freshwater, 2015), can be expanded to show that the sum of multiple analytical approaches yields more than its parts, even when a study employs an iterative hybrid approach with only one strand of data collection, such as the GTxA. In today’s technologically developing society, communication using textual data through digital tools plays an increasingly significant role in various domains of our social life. For example, text-based dialogue, as characterized by SNS, greatly affects our society and even transforms our way of living, for better or worse. Given this situation, the GTxA may help us uncover at a deeper level what is hidden in such text-based dialogue with its iterative hybrid approach to textual data, possibly contributing to achieving social justice (Mertens, 2007). Finally, we discuss challenges in using the GTxA. The use of the GTxA may be limited to a particular type of data and researchers with certain knowledge and skills. The grounded theory element of the GTxA assumes that researchers read the entire text or a portion of the relevant text in a particular topic; therefore, unlike big data analysis, the GTxA is limited to the analysis of a readable amount of textual data. Furthermore, researchers who apply the GTxA need to have knowledge and skills in operating text mining software. To analyse textual data efficiently, it is necessary to determine quickly the important topics hidden in huge amounts of data, as presented in Step 1 of the GTxA procedure. Therefore, the authors are currently exploring new methods involving linguistic

GROUNDED TEXT MINING APPROACH

statistical analyses and machine learning techniques that can be useful in extracting appropriate topic clusters.

Notes 1  This study is funded by the Japan Society for the Promotion of Science (JSPS) Grant-in-Aid for Scientific Research (B), (Grant number 20H03966). The members of the Japan Society for Mixed Methods Research (Kakai, H (P.I.); Abe, M.; Fukuda, M.; Hatta, T.; Inaba, M.; Inoue, M.; Kamei, T.; Kawamura, Y.; Makabe, S.; Narita, K.; Nozaki, M.; Ogawara, C.; Tajima, C.; and Takagi, A.) participated in the project. Drs. John Creswell and Michael Fetters served as advisers for this project. 2  For this chapter, we translated the results of our analyses conducted in Japanese into English. 3  https://khcoder.net/ 4  https://khcoder.net/en/ 5  www.screen.co.jp/as/products/monkin-main 6  A list of stop words is a set of frequently used words in a particular language. Lists are available for text mining to remove unimportant words. In our analysis, we used a stop words list published by Dr. Adam Crymble (https://gist.github.com/ acrymble/1065675).

WHAT TO READ NEXT Creamer, E. G. (2022). Advancing grounded theory with mixed methods. Taylor & Francis.

This textbook offers a detailed explanation of MM-GT methodology (MM-GTM) and fully integrated MM-GTM (FIMM-GTM). The textbook primarily benefits undergraduate and graduate students who wish to advance their skills in qualitative, mixed methods and evaluation research. The author offers definitions and explanations of terms related to both grounded theory and mixed methods, along with empirical examples using MM-GTM and FIMM-GTM. The author also provides a chapter on reporting. This textbook does not predicate its discussion on complex classifications of design types, enabling researchers to exercise their creative and innovative thinking in designing their research. Chapter 10 (this volume).

Vogl discusses how blurring boundaries between data classifications as qualitative or quantitative

343

requires new ways of thinking and approaching integration. This is a great introduction to three approaches to data integration: linkage, transformation, and consolidation. Poth, C. N., Bulut, O., Aquilina, A. M., & Otto, S. J. G. (2021). Using data mining for rapid complex case study descriptions: Example of public health briefings during the onset of the COVID-19 pandemic. Journal of Mixed Methods Research, 15(3), 348–373. https://doi.org/10.1177/155868982110 13925

This paper shows how text mining can contribute to the description of complex case studies in the unprecedented and changing context of the COVID19 pandemic. By integrating freely accessible data on the pandemic in Alberta (Canada), this study aims to gain new insights into how public health briefings can build credibility and trust within the rapidly evolving context. This paper also provides a table regarding definitions of text mining techniques and their advantages, along with examples of mixed methods research applications. This paper serves as an essential reference for the use of text mining in case study–mixed methods designs.

REFERENCES Bazeley, P. (2009a). Analyzing qualitative data: More than ‘identifying themes.’ The Malaysian Journal of Qualitative Research, 2(2), 6–22. Bazeley, P. (2009b). Editorial: Integrating data analyses in mixed methods research. Journal of Mixed Methods Research, 3(3), 203–207. https://doi. org/10.1177/1558689809334443 Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage. Biber, D., Conrad, B. S., & Reppen, R. (1998). Corpus linguistics: Investigating language structure and use. Cambridge University Press. Cain, L. K., MacDonald, A. L., Coker, J. M., Velasco, J. C., & West, G. D. (2019). Ethics and reflexivity in mixed methods research: An examination of current practices and a call for further discussion. International Journal of Multiple Research Approaches, 11(2), 144–155. https://doi. org/10.29034/ijmra.v11n2a2 Chang, T., DeJonckheere, M.,Vydiswaran, V. G. V., Li, J., Buis, L. R., & Guetterman, T. C. (2021). Accelerating mixed methods research with natural language processing of big text data. Journal of Mixed Methods Research, 15(3), 398–412. https:// doi.org/10.1177/15586898211021196 Charmaz, K. (2000). Grounded theory: Objectivist and constructivist methods. In N. K. Denzin & Y. S.

344

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Lincoln (Eds.), Handbook of qualitative research (2nd ed.). (pp. 509–535). Sage. Charmaz, K. (2014). Constructing grounded theory: A practical guide through qualitative analysis (2nd ed.). Sage. Charmaz, K., Thornberg, R., & Keane, E. (2018). Evolving grounded theory and social justice inquiry. In N. K. Denzin & Y. S. Lincoln (Eds.). The SAGE handbook of qualitative research (5th ed.). (pp.411–443). Sage. Creamer, E. G. (2018). Enlarging the conceptualization of mixed method approaches to grounded theory with intervention research. American Behavioral Scientist, 62(7), 919–934. https://doi. org/10.1177/0002764218772642 Creamer, E. G. (2022). Advancing grounded theory with mixed methods. Taylor & Francis. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (4th ed.). Sage. Fetters, M. D. (2020). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Sage. Fetters, M. D., & Freshwater, D. (2015). The 1+1=3 integration challenge. Journal of Mixed Methods Research, 9(2), 115–117. https://doi.org/10.1177/ 1558689815581222 Fetters, M. D., & Molina-Azorin, J. F. (2017). The Journal of Mixed Methods Research starts a new decade: The mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11(3), 291–307. https://doi. org/10.1177/1558689817714066 Glaser, B. G. (1978). Theoretical sensitivity. The Sociology Press. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. Aldine. Greene, J. C. (2015). Preserving distinctions within the multimethod and mixed methods research merger. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 606–615). Oxford University Press. Guetterman, T. C., Babchuk, W. A., Howell Smith, M. C., & Stevens, J. (2019). Contemporary approaches to mixed methods–grounded theory research: A field-based analysis. Journal of Mixed Methods Research, 13(2), 179–195. https://doi.org/10.1177/ 1558689817710877 Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Annals of Family Medicine, 13(6), 554–561. doi:10.1370/afm.186 Hammersley, M. (2018). Commentary—On the “indistinguishability thesis”: A response to Morgan. Journal of Mixed Methods Research, 12(3), 256–261. https://doi.org/10.1177/1558689818772764

Hesse-Biber, S. N. (2010). Mixed methods research: Merging theory with practice. Guilford Press. Hesse-Biber, S. N., Rodriguez, D., & Frost, N. A. (2015). A qualitatively driven approach to multimethod and mixed methods research. In S. N. Hesse-Biber & R. B. Johnson (Eds.). The Oxford handbook of multimethod and mixed methods research inquiry (pp. 3–20). Oxford University Press. Higuchi, K. (2016). KH Coder 3 Reference Manual. https://khcoder.net/en/manual_en_v3.pdf Hitchcock, J. H., & Onwuegbuzie, A. J. (2020). Developing mixed methods crossover analysis approaches. Journal of Mixed Methods Research, 14(1), 63–83. https://doi.org/10.1177/1558689819841782 Hoshino, A., & Suwa, M. (2019). A mixed methods study exploring difficulties in child care among Japanese mothers with depression. Asian Journal of Occupational Therapy, 15(1), 45–52. https:// doi.org/10.11596/asiajot.15.45 Inaba, M., & Kakai, H. (2019). Grounded text mining approach: A synergy between grounded theory and text mining approaches. In A. Bryant & K. Charmaz (Eds.), The SAGE handbook of current developments in grounded theory (pp. 332–351). Sage. Johnson, R. B., McGowan, M. W., & Turner, L. A. (2010). Grounded theory in practice: Is it inherently a mixed method? Research in the Schools, 17(2), 65–78. Kakai, H. (2022). Mixed methods research in the era of massive social change: A reflection on the 2021 MMIRA Asia Regional/7th JSMMR Conference. Annals of Mixed Methods Research, 1(1), 24–30. Maxwell, J. A. (2019). Distinguishing between quantitative and qualitative research: A response to Morgan. Journal of Mixed Methods Research, 13(2), 132– 137. https://doi.org/10.1177/1558689819828255 Mertens, D. M. (2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research, 1(3), 212–225. https:// doi.org/10.1177/1558689807302811 Morgan, D. L. (2018). Living within blurry boundaries: The value of distinguishing between qualitative and quantitative research. Journal of Mixed Methods Research, 12(3), 268–279. https://doi. org/10.1177/1558689816686433 Onwuegbuzie, A. J., & Combs, J. P. (2010). Emergent data analysis techniques in mixed methods research: A synthesis. In A. Tashakkori & C. Teddlie (Eds.), The Sage handbook of mixed methods in social and behavioral research (2nd ed., pp. 397– 430). Sage. Onwuegbuzie, A. J., & Hitchcock, J. H. (2015). Advanced mixed analysis approaches. In S. N. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 275–295). Oxford University Press. https://doi. org/10.1093/oxfordhb/9780199933624.013.1

GROUNDED TEXT MINING APPROACH

Pieterse, A. L., Lee, M., & Fetzer, A. (2016). Racial group membership and multicultural training: Examining the experiences of counseling and counseling psychology students. International Journal for the Advancement of Counselling, 38(1), 28–47. Plano Clark, V. L., Anderson, N., Wertz, J. A., Zhou, Y., Schumacher, K., & Miaskowski, C. (2015). Conceptualizing longitudinal mixed methods designs: A methodological review of health sciences research. Journal of Mixed Methods Research, 9(4), 297–319. https://doi.org/10.1177/1558689814543563 Plano Clark, V. L., & Ivankova, N. V. (2016). Mixed methods research: A guide to the field. Sage. Poth, C. N. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Poth, C. N., Bulut, O., Aquilina, A. M., & Otto, S. J. G. (2021). Using data mining for rapid complex case study descriptions: Example of public health briefings during the onset of the COVID-19 pandemic. Journal of Mixed Methods Research, 15(3), 348–373. https://doi.org/10.1177/155868982110 13925 Preissle, J., Glover-Kudon, R., Rohan, E. A., Boehm, J. E., & DeGroff, A. (2015). Putting ethics on the mixed methods map. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry. https://doi. org/10.1093/oxfordhb/9780199933624.013.46 Sandelowski, M. (2014). Unmixing mixed-methods research. Research in Nursing & Health, 37(1), 3–8. https://doi.org/10.1002/nur.21570 Sandelowski, M. (2018). A reluctant but necessary rebuttal to a rebuttal. Journal of Mixed Methods

345

Research, 12(4), 366–368. https://doi.org/10.1177/ 1558689818799427 Sandelowski, M., Voils, C. I., & Knafl, G. (2009). On quantitizing. Journal of Mixed Methods Research, 3(3), 208–222. https://doi.org/10.1177/15586 89809334210 Sherbino, J., Frank, J. R., & Snell, L. (2014). Defining the key roles and competencies of the clinician–educator of the 21st century: A national mixed-methods study. Academic Medicine, 89(5), 783–789. https://doi: 10.1097/ACM.0000000000000217 Stadnick, N. A., Poth, C. N., Guetterman, T. C., & Gallo, J. J. (2021). Advancing discussion of ethics in mixed methods health services research. BMC Health Services Research, 21(1), 1–9. https://doi. org/10.1186/s12913-021-06583-1 Strauss, A. L., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory (2nd ed.). Sage. VERBI Software. (2019). MAXQDA 2020 [computer software]. Berlin, Germany: VERBI Software. Available from maxqda.com. Vogl, S. (2019). Integrating and consolidating data in mixed methods data analysis: Examples from focus group data with children. Journal of Mixed Methods Research, 13(4), 536–554. https://doi. org/10.1177/1558689818796364 Wertsch, J. V. (1998). Mind as action. Oxford University Press. Yamauchi, K., Nakashima, M., & Nakao, M. (2021). What Japanese women with breast cancer decide: A mixed methods analysis of web-based openended responses. Asian Pacific Journal of Cancer Prevention, 22(9), 2909–2915. https://doi.org/ 10.31557/APJCP.2021.22.9.2909

24 A “Mixed Methods Way of Thinking” in Game-based Research Integrations Lisbeth M. Brevik

INTRODUCTION Games are ubiquitous. Game-based research— which involves studying the interactions between humans and video games—is emerging as an innovative area in which to advance discussions about the integration involved in mixed methods research designs. I use the term “video games” to capture a broad category involving console video games, mobile games, computer games, digital games or online games. Game-based research is concerned with various aspects of humans interacting with games and the effects of gaming on humans, whether serious games designed for education or commercial games designed for entertainment (Arnseth et  al., 2018; Gee, 2017; Schuster-Amft et  al., 2014). To conduct game-based studies, researchers draw upon methods and data across the various fields of education, e-learning, games development, educational psychology, information technology, computing science, multimedia, technology-enhanced education and beyond to enhance the user experience. The goal of game-based research is to make the user experience more engaging and, ultimately, more effective than a non-game experience (Barr & Copeland-Stewart, 2022; Brevik, 2022b; Scholtz, 2022).

The distinction between a game-based experience and gamification is important; whereas the former involves the use of video games for learning or entertainment (Arnseth et al., 2018; Brevik, 2019; Scholtz, 2022), the latter refers to the use of game principles as incentives in a non-game environment (Costa et  al., 2021; Cruaud, 2018). Another useful distinction is that of game-based experiences and virtual reality (VR): VR involves the use of interactive simulations to engage in environments similar to real-world situations and has its origins in the gaming industry (SchusterAmft et al., 2014; Warland, 2018). In other words, there can be games in gamification and VR, but not all gamification or VR includes games. Integration has conceptually redefined the field of mixed methods research; hence, the purpose of the current chapter is to advance the conversation on integration by providing a real-world illustration of how researchers apply a “mixed methods way of thinking” (Greene, 2007) to enhance the integration potential for innovation through mixed methods designs. I describe the prevalence of and potential for game-based research integrations, the design implications of framing game-based research within Greene’s (2007) theoretical framework, and an empirical example illustrating the potential of the integration purpose of innovation.

A “MIXED METHODS WAY OF THINKING” IN GAME-BASED RESEARCH INTEGRATIONS

Then, I provide access to the methodological and ethical decisions behind the design and conclude the chapter with some implications for innovative integration in game-based research.

PREVALENCE OF AND POTENTIAL FOR GAME-BASED RESEARCH INTEGRATIONS Scholars have shown why focusing on game-based research is warranted across various fields (Arnseth et al., 2018; Brevik & Buchholtz, 2022; Campbell et  al., 2019; Gee, 2017; Gorbanev et  al., 2018; Rüth & Kaspar, 2021; Scholtz, 2022; SchusterAmft et  al., 2014). Although most game-based studies are either quantitative or qualitative, gamebased studies increasingly turn to mixed research to understand complexity. As such, integration occurs not only with qualitative and quantitative data, but also as monomethod integration within either of these data types (Schoonenboom, 2022). In the neuroscience fields, a game-based study used brain imaging to test the hypothesis that action games (e.g. Halo, Counterstrike, Gears of War, Call of Duty) increase selective attention in gamers compared with non-gamers (Bavelier et al., 2012). The researchers integrated quantitative survey data in phase 1 to identify gamers and nongamers, with the reaction time in response to different stimuli within and across the groups in phase 2, thereby integrating quantitative data only. A systematic review of digital games on learning that included 69 unique study samples indicated that digital games significantly enhance student learning relative to non-game conditions (Clark et  al., 2016), and because of the focus on making causal inferences regarding game-based effects, only studies using randomized controlled trial and controlled quasi-experimental designs were included, thereby prioritizing quantitative integration. Similarly, a game-based study in language education prioritized qualitative integration to investigate the significance of connecting informal and formal language learning through online gaming (Brevik & Holm, 2022). The researchers integrated qualitative observations (video and screen recordings) with qualitative self-reports (interviews, documents), finding through integration that connecting game-based learning in the classroom with students’ online gaming outside school developed language identity and proficiency. Still, it should come as no surprise that the “integration of qualitative and quantitative data is a quintessential characteristic of mixed methods research” (Mertens, 2022, p. 71). If the research focus is on game effectiveness or user experience, then the integration of different data types

347

might be warranted to capture not only quantitative information about in-game user choices or effects on learning outcomes, but also qualitative user experiences and opinions. There has been a call for the integration of both data types in game-based research (Borg & Al-Busaidi, 2012; Brevik, 2016, 2019, 2022b; Brevik & Buchholtz, 2022; Schwarz, 2020). In line with this argument, a systematic review of research on serious games in medical education found that among the 21 included studies, both quantitative effect studies and mixed methods designs were identified (Gorbanev et  al., 2018). However, some studies combined effect data with open-ended questions or focus groups, with little information about integration, hence making it challenging to identify whether integration in these studies “means more than meeting a definition of mixed methods research” (Guetterman et  al., 2020, p. 430). Conversely, in a mixed methods study comprising a survey with closed and open questions, the integration of both data types was considered essential in all aspects of data collection, analysis and interpretation to identify the “significance of video games and the potentially positive nature of games’ effects on well-being” (Barr & CopelandStewart, 2022, p. 93). In these examples, purpose is key. Greene (2007) stated: “Methodology is ever the servant of purpose, never the master” (p. 97). The mixed methods research field can benefit from focusing on integration (Hitchcock & Onwuegbuzie, 2022), and many researchers have relied on multiple purposes to explain why research problems require mixed integration (Collins et  al., 2006; Greene et  al., 1989; Poth, 2018). Of particular interest to game-based research is Poth’s (2018) integration typology: corroboration, completion, explanation, development, infusion or innovation. In the following, I weave in a discussion of the typologies through examples from various fields of game-based research to illustrate integration purposes. These examples demonstrate the prevalence of game-based integrations. Then I discuss the untapped potential for the mixing purpose of innovation through an empirical example of integration in the field of game-based research.

Corroboration: When a Single Data Type is Insufficient When a single data type is insufficient, the need to integrate the data with a complementary data source for corroboration offers further evidence of divergence or convergence (Poth, 2018). An example is a German study investigating how to

348

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

use commercial video games in the educational setting of biology and history lessons in high school (Rüth & Kaspar, 2021). To corroborate the students’ experiences with the game and their learning of topical knowledge through gaming, the authors integrated qualitative data (audiorecorded lessons) with quantitative data (pre- and post-test) and mixed data (survey data with open and closed questions). The authors stated triangulation as the purpose of using a mixed methods design, by combining quantitative and qualitative data. According to Poth (2018), “a purpose for integration stated as triangulation [… ] can serve the purpose of either corroboration or completion” (p. 37).

Completion: When Additional Data Type Enhances Evidence Another purpose of integration is completion, when an additional data type is needed to enhance evidence (Poth, 2018). In applied computing, a European project aimed to reflect on hate speech online, suggesting possible ways to combat it through serious games (Costa et  al., 2021). Starting from a quantitative survey among young people in Portugal, Italy and Lithuania, the researchers found that survey data were insufficient for explaining the participants’ perspectives on the role of games in reducing hate speech; they suggested the integration of qualitative data (online focus groups) in an exploratory phase, to better understand the quantitative findings. After conducting focus groups, the researchers identified a need for further data completion and suggested participatory research to involve young people in developing games. Here, the prevalence of game-based integration suggested by the need for additional data types helped explore the meanings measured by the statistics (survey) as a path to mutual understanding (focus groups) to promote tolerance, thereby reducing hate speech by developing games (participatory research).

action. They recruited educators who were students of educational technology, instructional design, or e-learning. The researchers embedded individual phone interviews and face-to-face focus group interviews within the context of a prepost survey design. The purpose was to explain the usefulness of a rubric in their evaluation of mobile apps and web-based games for use in teaching. All data types (quantitative and qualitative) were collected concurrently, and the researchers analyzed the data separately. Then, the qualitative and quantitative data were integrated through interpretation and reporting for explanation purposes. The interviews helped explain the educators’ experiences with the rubric and integration of the data types, providing recommendations for its usefulness. As such, the qualitative interviews helped explain the quantitative survey results of the game-based study.

Development: When Protocols Facilitate Access to Further Data or Populations Development concerns creating an instrument, such as a questionnaire or interview guide, or to use a data source for sampling purposes to gain access to a population that can otherwise be difficult to access (Poth, 2018). For example, in Austria, Schwarz (2020) designed a sequential explanatory study in education, in which her initial quantitative data informed the subsequent development of a qualitative interview guide. The purpose of integration was to access more data on the relationship between engagement with English outside school, including online gaming and vocabulary knowledge measured in school. Data on the frequency and amount of secondary students’ out-of-school engagement with English were collected through questionnaires, language diaries and vocabulary tests. The quantitative data were used to develop an interview protocol, and student focus groups were carried out to understand their perspectives more fully.

Explanation: When Additional Data Type Expands Understanding of the Findings

Infusion: When a Theoretical Stance Transforms Perspectives

The need for explanation is yet another purpose of integration, such as when a research problem involving qualitative data expands the initial quantitative findings (Poth, 2018). For example, in the USA, Campbell et  al. (2019) evaluated mobile apps and web-based games to inspire community

Infusion is when a study seeks to advocate for social change (Poth, 2018). One example is Campbell et al.’s (2019) rubric for the evaluation of games for social change to identify whether a game had the potential to inspire community action. Another example is Costa et  al.’s (2021)

A “MIXED METHODS WAY OF THINKING” IN GAME-BASED RESEARCH INTEGRATIONS

study on how to combat hate speech online. In both studies, the researchers integrated qualitative and quantitative data to understand whether online games had the potential to increase the participants’ awareness of social issues and transform perspectives to advocate for social change. As such, they used quantitative survey measures to identify the potential in developing games for social change and gained participants’ perspectives on these measures through interviews, focus groups or participatory research.

Innovation: When Complexity Demands the Integration of Both Data Types to Generate What is Yet to be Known The final purpose of integration is innovation. Poth (2018) argued that innovation in mixed methods research is necessary to address uncertainty, whether concerning methodological or contextual conditions, making “the mixing purposes for innovation […] yet to be specified” (p. 39). Hence, the purpose of innovation might relate to the complexity of designing an appropriate mixed integration. Although the above studies illustrate one or two of Poth’s (2018) typologies, studies might identify numerous purposes of integration. The presence of multiple purposes promotes fuller integration because both quantitatively and qualitatively oriented purposes will inform the overall conceptualization of the study (Onwuegbuzie et al., 2018). The integration purposes in the above game-based research studies have theoretical implications that align with “a mixed methods way of thinking” (Greene, 2007), particularly in the emphasis on integration to capture complexity and multiple ways of knowing.

DESIGN IMPLICATIONS OF THE THEORETICAL FRAMEWORK: “A MIXED METHODS WAY OF THINKING” Greene (2007) crafted a theoretical framework that values not only various voices and perspectives, but also the differences between them: A mixed methods way of thinking aspires to better understand complex social phenomena by intentionally including multiple ways of knowing and valuing and by respectfully valuing differences. (p. 17)

349

Greene (2007) defined the study of complex social phenomena as one that involves not just one or more individuals, but also the individuals and their context. As such, game-based research involves games and individuals, including individuals interacting with games and each other, in-game contexts and contexts surrounding the games. Thus, the mixed methods way of thinking resonates with the need to explore and deepen our understanding of the complex social phenomena characterizing the field of game-based research and includes the premise that gaming takes place in various physical and virtual spaces (Gee, 2017). However, a deliberate integration of both data types is necessary to capture these voices and perspectives, which is in line with Greene’s (2007) theoretical framework.

INNOVATIVE INTEGRATION AND AN ILLUSTRATIVE EXAMPLE: THE VOGUE STUDY To illustrate how game-based research can benefit from innovative integration, I present an empirical study in which I brought mixed methods into the home of gamers. The study is part of the VOGUE project at the University of Oslo, Norway. The research aim was to identify how and why the use of English outside school developed Norwegian teenagers’ proficiency in English. Understanding teenagers’ language development would require, among other things, making sense of their language use on a daily basis and designing strategies for capturing not only their own perspectives, but also researchers’ observations of such language use across contexts, including in the home (Beiler et al., 2021). Teenagers in Norway are among the most proficient users of English as a second language in Europe (Education First, 2022). Along with their Scandinavian neighbours, Norwegians are widely exposed to English, particularly online, and 98 per cent of Norwegians have internet access compared with the global average of 67 per cent (Internet World Statistics, 2022). In this context, Brevik (2019) identified three commonplace language profiles relating to teenagers’ interests outside of school: “gamers” who use English for online gaming, “surfers” who use English to find information on the internet and “social media users” who use English on social media platforms. The literature has shown that game-based entertainment has the potential to develop language proficiency to a greater extent than other uses of English (Gee, 2017). However, little is known about the

350

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

characteristics of these teenagers’ gameplay from an in-game perspective. This research gap was addressed in the VOGUE study. The innovative integration introduced here relates to the valuing of perspectives and voices across contexts and consists of 10 strategies: (1) a mixed methods way of thinking; (2) students as co-researchers; (3) innovation purpose of integration; (4) integrated hybrid design; (5) facilitating access to key population; (6) stepwise data collection; (7) integrated data analysis; (8) data visualisation; (9) unlocking metaphor; and (10) ethical consideration. Table 24.1 introduces the ten strategies. The ten strategies build on and extend Fetters and MolinaAzorin’s (2017) dimensions of integration, which span the mixed methods research process “to help researchers think about integration broadly to achieve fully integrated mixed methods research” (Guetterman et al., 2020, p. 431).

Integration Strategy 1: A Mixed Methods Way of Thinking The mixed methods way of thinking refers to how researchers orient their research towards philosophical and theoretical assumptions to capture multiple voices and perspectives. Believing that multiple philosophical and theoretical stances can be combined, I oriented the VOGUE study towards Greene’s (2007) theoretical framework, which resonates with the philosophical mindset of dialectical pluralism (Johnson, 2017). Dialectical pluralism builds on the belief that reality is complex and that there is value in capturing multiple perspectives and diverse ways of knowing, where

qualitative information represents one type of knowing and quantitative information represents another. I carefully planned to capture diverse voices (i.e., gamers and non-gamers) and diverse viewpoints (i.e., the emic perspective of gamers and non-gamers and the etic perspective of researchers) in different contexts (i.e., at school and home). Moreover, I planned to pursue contradictory or unexpected results if they occurred (e.g., planning for the unexpected) and analyze opposing views by placing them into a dialectic interaction to create new syntheses concerning the connection between online gaming and language development.

Integration Strategy 2: Students as Co-researchers The co-research strategy builds on the belief of involving not only experienced researchers, but also students as co-researchers, ensuring the team comprises diverse voices and perspectives. In the co-research strategy, Master’s students are invited into ongoing research projects in innovative ways (Brevik, 2020, 2022a; Brevik et  al., 2022; Eriksen & Brevik, 2022). In VOGUE, coresearchers were engaged from the onset, participating in workshops to learn how to use the research tools and procedures established in VOGUE, and in planning, data collection and analysis. Each co-researcher was trained to become an expert of one data type, to promote the team’s capacity to pursue mixed methods questions and to have a mutual understanding of the game-based study. Some co-researchers were gamers themselves, and suggested the choice of

Table 24.1  Ten strategies for innovative mixed methods integrations: valuing diverse voices and perspectives across contexts Integration strategies

Integration dimensions

1. A mixed methods way of thinking 2. Students as co-researchers 3. Innovation purpose of integration 4. Integrated hybrid design 5. Access to key population 6. Stepwise data collection 7. Integrated data analysis 8. Data visualization 9. Unlocking metaphor 10. Ethical consideration

Philosophical and theoretical dimensions Researcher and team dimensions Literature review, rationale and study dimensions Research design dimension Sampling dimension Data collection dimension Data analysis dimension Interpretation and dissemination dimensions Rhetorical dimension Research integrity dimension

Source: Author created.

A “MIXED METHODS WAY OF THINKING” IN GAME-BASED RESEARCH INTEGRATIONS

screen recording software, the creation of routines for software installation in the gamers’ homes, and trustworthiness in recruiting gamers (Darras, 2021). As such, the co-research strategy contributed invaluable expertise to the research team concerning several game-based aspects.

Integration Strategy 3: Innovation Purpose of Integration I chose the innovation strategy of integration (Poth, 2018), because of a need to address what was yet unknown about teenagers’ language use during gaming. Based on the research gaps, the innovation strategy provided opportunities to address uncertainty and complexity in the VOGUE study concerning methodological and contextual conditions. Methodologically, quantitative and mixed methods studies among teenagers in Norwegian secondary schools found connections between extensive online gaming and English use during gameplay (Brevik, 2016, 2019; Brevik & Hellekjær, 2018; Sletten et  al., 2015). Although the studies complemented each other, they combined language proficiency measures, school grades and self-reports, with no observation data. Thus, contextually, there was a need for innovation by bringing mixed methods into gamers’ homes for observation in naturally occurring gaming situations. It further justified the need for integration of perspectives that could be captured through mixed data types, aiming for an in-depth understanding of how teenage gamers used languages during gaming and why they believed it was important for developing their English proficiency. I developed the following research question: “How and why do teenagers use English during gameplay in naturally occurring contexts in the home?”

Phase 1: self-report (qualita�ve and quan�ta�ve)

Integration strategy 4: Integrated Hybrid Design Based on the need for innovation, I created an integrated hybrid design for the VOGUE study at the design level—the conceptualization of the study—as defined by Tashakkori et  al. (2020)— because it allows for a mixture of elements from parallel, sequential and conversion designs. The design integrated the various voices and perspectives necessary to capture language use in gamebased contexts. Figure 24.1 illustrates the initial hybrid design, comprising a parallel design in the first phase and sequential design across all phases, including a potential phase 3 in case of unexpected outcomes from phases 1 and 2 (emergence, see Brevik, 2022b; Schoonenboom, 2022). Phase 1 integrated the participants’ voices through selfreports (emic perspective), phase 2 emphasized the researchers’ observations (etic perspective), and phase 3 was open to any perspective. Figure 24.1 illustrates the parallel design in phase 1 where several circles indicate full integration within the phase—of methods, voices, perspectives and inferences—before integration with phase 2 observations. Integration between phases 1 and 2 is illustrated using an arrow, and the integration with phase 3 uses two triangles to illustrate a keyhole that might need to be unlocked. Figure 24.2 illustrates the final hybrid design, comprising three phases. Phases 1 and 2 were conducted according to plan during three consecutive weeks. Phase 3 was the result of unexpected outcomes across the first two phases and was conducted six months later. Thus, phase 3 can be considered unlocked by phases 1 and 2, serving as a key (see strategy 9). The main difference between the initial and final designs was the confirmation that the third phase was necessary. The complexity of these figures depicts the innovation purpose of integration. Although phases 1 and 2 were

Phase 2: observa�on (qualita�ve)

Phase 3: if needed (planning for the unexpected)

Figure 24.1  The initial hybrid design (two planned phases and an optional third phase) Source: Author created.

351

352

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Phase 1: self-report (corrobora�on)

Phase 2: observa�on (comple�on)

Phase 3: self-report and observa�on (explana�on)

Figure 24.2  The final hybrid design (three phases) Source: Author created.

informed by the aforementioned purposes from Poth’s (2018) typology—corroboration among self-reported data types in phase 1 and completion in phase 2 by adding observation as a new data type to enhance evidence—phase 3 was informed by explanation, seeking to extend the breadth and range of enquiry by repeating both self-reported data (quantitative) and observation (qualitative).

Integration Strategy 5: Access to the Key Population Using this integration strategy facilitated access to the key population—namely, gamers and nongamers. The VOGUE study used two basic types of sampling procedures: purposive and convenience (Fetters & Molina-Azorin, 2017; Tashakkori et  al., 2020). Based on the literature review, the key population for the VOGUE study was defined as teenage gamers in secondary school. The team used purposive sampling, approaching English teachers in secondary school who were willing and able to participate in the study. This procedure facilitated access to two male-dominated vocational classes taught by the same teacher, in which several students were gamers. However, to capture various voices, we also facilitated access to nongamers in the same context. To this population, we used convenience sampling by approaching nongamers in the sampled classes. In total, 34 students across both classes were willing and able to participate, which was appropriate for uncovering how they used English outside school, including, but not exclusively, through online gaming. Figure 24.3 shows that in phase 1, we used an identical mixed methods strategy, where qualitative and quantitative data were collected among the entire sample (gamers and non-gamers), and in phase 2, we used a nested mixed methods strategy (Fetters & Molina-Azorin, 2017; Tashakkori et  al., 2020),

where the qualitative data were collected from a subgroup (gamers). Thus, the gamers (n = 4) were intentionally sampled for screen recordings based on specific gamer profile characteristics (Brevik, 2019) from all 34 participants in phase 1. In phase 3, we returned to the identical sample to collect new quantitative and qualitative data. The added value of integrating the samples was threefold. First, we gained access to the voices of both gamers and nongamers. Second, we collected their perspectives as both qualitative and quantitative information twice (phases 1 and 3). Finally, the nested strategy facilitated access to the VOGUE study’s key population of male gamers (phase 2).

Integration Strategy 6: Stepwise Data Collection The integration strategy of stepwise data collection emphasizes mixed methods research procedures to integrate different types of data collection within one phase before integration across phases. The joint display in Figure 24.4 illustrates VOGUE’s stepwise collection of three qualitative data types (interview, school text and screen recording) and three quantitative ones (survey, language log and language frequency). In phase 1, we collected four self-reported data sources (interview, school text, survey and language log) among gamers and non-gamers concerning their use of English beyond the school context. For integration, we matched closed questions in the survey about the use of English with open questions about the use of English in the school text and interviews; similar closed questions were used in the language log that they filled out daily for two weeks. In phase 2, the key population of male gamers provided one source of observation data (screen recordings), capturing gameplay in the home, including numerical data (language

A “MIXED METHODS WAY OF THINKING” IN GAME-BASED RESEARCH INTEGRATIONS

353

Phase 1 and 3: identical mixed methods strategy to access gamers and non-gamers (N=34)

Phase 2: nested mixed methods strategy involving male gamers only (n=4)

Figure 24.3  The integrated sampling strategy to gain access to key populations Source: Author created.

Quantitative

Qualitative Interview

School text

Survey

Phase 1

Phase 1

Language log

Phase 1

Phase 1

Language frequency

Screen recording Phase 3 Phase 2

Figure 24.4  Joint display: integration through stepwise data collection within and across phases Source: Author created.

frequency). The observation of the gameplay was captured through video and chat script recorded on their computer screens (Figure 24.5), integrated with audio recording of the oral chat. The recordings also provided opportunities to measure frequency of English use, thereby contributing mixed data collection. In phase 3, we returned to the entire sample for the collection of a new round of self-reports (language log) and observation data (screen recording), where all participants recorded a YouTube video to demonstrate an English language activity reported in the language logs. As such, the stepwise data collection strategy aligned with Greene’s (2007) definition of the study of complex social phenomena that addresses the individual and context, to value the perspectives of both gamers and non-gamers in the various contexts they inhabit.

Integration Strategy 7: Integrated Data Analysis The integrated data analysis strategy emphasizes integration within and across each phase of a

mixed methods study (Fetters & Molina-Azorin, 2017; Onwuegbuzie et al., 2018; Shannon-Baker, 2022; Tashakkori et al., 2020). We used quantitative integration of quantitative data (phases 1 and 3), qualitative integration of qualitative data (phase 2) and integration through the quantitizing of qualitative data (transforming qualitative data into quantitative data; phases 1–3). In phase 1, we matched closed-ended questions in the survey and language logs about the use of English (quantitative analysis of quantitative data) and open-ended interviews and school text questions about the use of English (quantitizing of qualitative data). We created a bar chart figure that displayed the integration of the four data sets (Figure 24.7), presenting the qualitative data in the two bars on the left and the quantitative data in two bars on the right. Integrating the teenagers’ responses across data types corroborated the self-reports and showed that all participants used English for online gaming. In phase 2, the co-researchers transcribed the oral chat from the screen recordings for integrated data analysis. We thematically analyzed the transcriptions, identifying the languages being used

354

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 24.5  Screen capture of the gameplay from one screen recording Source: Author created. Note. The script is removed for anonymity.

(qualitative analysis of the qualitative data). The qualitative analysis showed that although the gamers frequently used English, they also used Norwegian and even Gamerspeak (gaming terminology) in oral chat (Darras, 2021). We used transcriptions to accumulate the total number of words in each language (quantitizing of the qualitative data) and, through quantitizing, it became clear that Norwegian was used the most, even though there was seldom a minute that passed without any English being spoken in the game (Darras, 2021). The two data sets in phases 1 and 2 intertwined and informed each other and, through integration,

we verified divergence between the gamers’ actual language use (Norwegian mainly) and their reported language use (English mainly). Thus, the integrated analysis provided a more complete picture of the gamers’ language use during gameplay than each phase separately. Expanding the design with phase 3 helped make inferences about the use of English in the identical sample. We found convergence of the findings in the language logs across phases 1 and 3, indicating that both gamers and non-gamers reported using English mainly. However, we found divergence in the screen recordings across Qualitative

Interview

School text

Screen recording

Quantitative

Survey

Language log

Phase 1: Quantitative integration of quantitative data; quantitizing of qualitative data

Language frequency

Phase 3: Quantitative analysis; quantitizing of qualitative data Phase 2: Qualitative analysis; quantitizing of qualitative data

Figure 24.6  Joint display: integrated data analysis within and across phases Source: Author created. Note. Arrows indicate across phase integration.

355

A “MIXED METHODS WAY OF THINKING” IN GAME-BASED RESEARCH INTEGRATIONS

Log

Survey*

96

Interview

ONLINE GAMES*

SOCIAL MEDIA*

INTERNET*

SERIES & FILMS

MUSIC

BOOKS

0

0

0

0

0

17 13 11

31

31

38

56

67

81

81

89

89 90

96

100 87

96

81

89 93

100 100

Texts

FACE-TOFACE

Figure 24.7  Percentage of participants (n = 34) who reported using English * The survey included only three categories. Source: Author created.

phases 2 and 3, demonstrating less variation in the use of English in the phase 3 YouTube videos than in the phase 2 gameplay. Here, English was used more consistently in the specific task they were asked to provide in phase 3, whereas they drew on their language resources (i.e., English, Norwegian, Gamerspeak) more freely when gaming at home. The impression of online gaming as their main source of English language development (Phase 1) and impression of the gamers using both Norwegian and English extensively for online gaming (Phase 2) were nuanced, with context awareness as another major explanation for how and why they used English outside school (phase 3). This game-based research helped demonstrate the general utility of innovative mixed methods design integrations.

Integration Strategy 8: Data Visualization Data visualization emphasizes the value of creating visuals to display complex mixed methods designs throughout the research process, and a means for sharing study findings (Shannon-Baker, 2022). For a discussion of using visuals to teach and learn mixed methods research, see also Chapter 23 (this volume). In the VOGUE study, I

demonstrated how a table visual could be used to show integration strategies in the game-based study (Table 24.1). I further illustrated the potential of adding a new phase to the research design (Figures 24.1 and 24.2), elucidating how the understanding of the hybrid design changed over time. I illustrated how to integrate identical and nested samples (Figure 24.3) and created a joint display to show the integration of qualitative and quantitative data (Figure 24.4). To show the in-game context, I included a screen capture from the gameplay (Figure 24.5). I created another joint display to show integrated analyses within and across phases, including quantitizing integration (Figure 24.6). Finally, I created a bar chart-style joint display to visualize the integration of the findings across four data sets (Figure 24.7).

Integration Strategy 9: Unlocking Metaphor The unlocking metaphor indicates how divergence in the data might serve as key to unlocking a new research phase, thereby addressing the unknown. “Unlocking” is a term increasingly used in computer science to describe a process where a password or other form of authentication

356

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

provides access to data, typically with a key. I adapted the existing terminology of “unlocking” to illustrate the need for innovative integration (Poth, 2018). I have used the term “unlocking” before (Brevik, 2022b) to represent the final step (i.e., the key) in identifying the need to initiate new research. This is relevant in hybrid designs involving more than one research phase, “in terms of data collected and analysed, points of integration, inferences, emergent outcomes influenced [the] subsequent phase” (Brevik, 2022b, p. 202). In the VOGUE study, the divergent results across phases 1 and 2 unlocked phase 3. The unlocking feature involves three points of integration. First, I identified divergence between phases 1 and 2. Second, I realized the need to initiate phase 3 to explain the divergence. Third, I realized that the divergence between the reported and observed English use during gameplay was, in fact, not unexpected because it was a result of the natural context outside school. Thus, the unlocking feature of phase 3 made the design suitable for game-based mixed methods research.

Integration Strategy 10: Ethical Consideration The strategy of ethical consideration is imperative for mixed methods research among human participants, where the collection of several data sources across contexts increases the risk of collecting

unwanted personal data (NESH, 2019). Because the game-based research in the VOGUE study stretched into the private sphere of homes, it gave rise to increased ethical considerations. First, to ensure privacy and avoid collecting unnecessary personal data, we asked the gamers to turn off the integrated cameras and record their voice and screen only. On the screen, personal data included screen captures with the gamers’ names or nicknames. Hence, we instructed them to pause the recording in situations where personal data were visible, and they were given the choice of whether to submit the screen recordings (Ho, 2019). Such procedures enhanced “processual consent”, meaning consent that is actively renewed during data collection (Sieber & Tolich, 2013). Finally, we instructed the co-researchers to bring the recording to the university data lab immediately after receiving it from the gamers to remove or blur any personal information about the gamers or a third party in the recordings. To further consider third parties, we gave each gamer a handwritten note with instructions (Figure 24.8). We asked them to inform any player in the game about the recording and ask for their consent before screen recording; if not, they would also turn off the visual screen recording and only record their own voice. They could provide our email to other players who wanted information and to place a note on the door to protect family members by informing them about the recording. We also asked the gamers to use a headset so that if anyone came into the gaming room, their voices

Rules 1. Inform about recording! Get consent, if not, turn off the screen recording and record your own voice only. 2. If you play with a group, inform them about the recording and ask for their consent. Give them my email if any questions. 3. Place a note on your door saying that recording is in progress. 4. Inform your parents/guardians. Thank you!

Figure 24.8  A written, step-by-step procedure to protect the third party Source: Author created.

A “MIXED METHODS WAY OF THINKING” IN GAME-BASED RESEARCH INTEGRATIONS

would not be recorded. Finally, we reminded them to inform their parents about the recording for the VOGUE study.

IMPLICATIONS AND FUTURE DIRECTIONS FOR GAME-BASED RESEARCH DESIGN AND BEYOND At the core of game-based research are complex phenomena (Gee, 2017). This resonates with Greene’s (2007) notion of a mixed methods way of thinking: research that aims to capture multiple perspectives because it involves not only humans, but humans and the context. The notion of multiple perspectives applies to one or more participants and to one or more game-based contexts across the various fields of education, e-learning, games development, educational psychology, information technology, computing science, multimedia, technology-enhanced education and beyond. Mixed methods integrations, which are particularly suitable for complex and multilayered topics, promise to provide additional insight within the assumedly multilayered and complex field of game-based research (Borg & Al-Busaidi, 2012; Brevik, 2016, 2019, 2022b; Brevik & Buchholtz, 2022; Gorbanev et al., 2018; Schwarz, 2020). The new perspectives that this chapter brings concern the introduction of ten strategies for innovative mixed methods integrations that can be applied to game-based research. In this vein, mixed methods integrations add value to the complex phenomena that characterize the field of game-based research. However, mixed methods integrations do not always lead to complementary or convergent findings. The results may be incoherent, or the qualitative and quantitative results might diverge or even contradict each other (Onwuegbuzie et al., 2018; Poth, 2018; Tashakkori et al., 2020). This identification of divergence or even paradox in the material might be an important methodological contribution (Brevik & Buchholtz, 2022). The challenge is to gain additional value from the innovative integration with regard to the overarching research question, to “produce a whole through integration that is greater than the sum of the individual qualitative and quantitative parts” (Fetters & Freshwater, 2015, p. 116). Such an effort is necessary if we want to do justice to the complexity of game-based research integrations. The integration approach of valuing diverse voices and perspectives across contexts presented in this chapter could be useful not only for game-based research, but also for the mixed

357

methods community. The innovation approach could be applied more generally in other research fields concerned with complex social phenomena (Greene, 2007), to address other questions than game-based ones, and thereby contribute to the integration discussion more broadly. In particular, the VOGUE study example demonstrated how mixed methods research can be used to apply mixed methods integration practices, relevant for studies that use both qualitative and quantitative data that represent emic and etic perspectives among various participant groups (Beiler et  al., 2021; Brevik & Holm, 2022). Key considerations for researchers deciding to use the ten integration strategies, include (1) theoretically, considering how to apply a “mixed methods way of thinking” to their study; (2) empirically, discussing the benefits and challenges of inviting students as co-researchers into the mixed methods research team; (3) methodologically, attending to mixed methods integration within each phase before integration across phases; and (4) ethically, attending to research integrity to ensure participant privacy and consideration of third parties throughout the research integrations. Guetterman et  al. (2020) have convincingly argued that researchers need to attend to integration broadly to capture the complexities of mixed methods integration. Future directions for gamebased research design could therefore attend to the integration potential for innovation through mixed methods designs. Integration, in the mixed methods sense of the word, is mainly associated with the integration of qualitative and quantitative data and analyses. Such integration does not necessarily fit all game-based research purposes, nor incorporate all relevant types of integration (Fetters & Molina-Azorin, 2017). In this chapter, I therefore argue the need for a mixed methods way of thinking that is relative to the game-based contexts where qualitative and quantitative information is articulated. Rather than taking the mixed methods integration for granted as the only rational means of research integration, I refer to other types of integration. Bavelier et al. (2012) and Clark et al. (2016) have, for instance, used quantitative integration of quantitative data, whereas Brevik and Holm (2022) used qualitative integration of qualitative data. They provide integration of a different kind. Using such studies as a starting point, this chapter provides examples of how to change the integration approach from a monomethod one to mixed methods approaches to innovative game-based integrations. Adoption of a mixed methods way of thinking (Greene, 2007) can raise new questions and encourage the methodological innovations needed to solve increasingly complex problems concerning game-based research integrations,

358

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

but it also has an inherent pragmatic orientation towards linking the supposedly incompatible.

WHAT TO READ NEXT? Creamer, E. G. (2018). An introduction to fully integrated mixed methods research. Sage.

For readers interested in how and when full integration can take place in mixed methods research. Poth, C. N. (2018). Innovation in mixed methods research. A practical guide to integrative thinking with complexity. Sage.

For a practical guide to innovation in mixed methods. Plass, J. L., Mayer, R. E., & Homer, B. D. (2019). Handbook of game-based learning. The MIT Press.

For a deeper understanding of game-based research studies and designs, including qualitative, quantitative and mixed methods research.

REFERENCES Arnseth, H. C., Hanghøj, T., & Silseth, K. (2018). Games as tools for dialogic teaching and learning. In H. C. Arnseth, T. Hanghøj, T. Henriksen, M. Misfeldt, R. Ramberg, & S. Selander (Eds.), Games and Education: Designs in and for Learning (pp. 123–139). Brill Academic Publishers. https:// doi.org/10.1163/9789004388826_008 Barr, M., & Copeland-Stewart, A. (2022). Playing video games during the COVID-19 pandemic and effects on players’ well-being. Games and Culture, 17(1), 122–139. https://doi.org/10.1177/155541 20211017036 Bavelier, D., Achtman, R., Mani, M., & Föcker, J. (2012). Neural bases of selective attention in action video game players. Vision Research, 61, 132–143. https://doi.org/10.1016/j.visres.2011.08.007 Beiler, I. R., Brevik, L. M., & Christiansen, T. (2021). Skjermopptak som forskningsmetode i og utenfor klasserommet [Screen recording as research method in and outside the classroom]. In E. Andersson-Bakken & C. P. Dalland (Eds.), Metoder i klasseromsforskning. Forskningsdesign, datainnsamling og analyse [Methods in classroom research. Research design, data collection and analysis] (pp. 239–260). Scandinavian University Press. Borg, S., & Al-Busaidi, S. (2012). Learner autonomy: English language teachers’ beliefs and practices (ELT Research Paper 12-07). British Council. www.

teachingenglish.org.uk/sites/teacheng/files/b459% 20ELTRP%20Report%20Busaidi_final.pdf Brevik, L. M. (2016). The gaming outliers: Does outof-school gaming improve boys’ reading skills in English as a second language? In E. Elstad (Ed.), Educational technology and polycontextual bridging (pp. 39–61). Springer. Brevik, L. M. (2019). Gamers, surfers, social media users: Unpacking the role of interest in English. Journal of Computer Assisted Learning, 35, 595– 606. https://doi.org/10.1111/jcal.12362 Brevik, L. M. (2020, 2 September). Studenter som medforskere. Hvordan involvere masterstudenter i forskningsprosjekter og som medforfattere? [Students as co-researchers. How to involve master’s students in research projects and as co-researchers?]. Education Prize Lecture. University of Oslo. Brevik, L. M. (2022a). Medforskning i lærerutdanningen [Co-research in teacher education]. Bedre Skole, 1, 52–57. Brevik, L. M. (2022b). The emergent multiphase design: Demonstrating an integrated approach in the context of language research in education. In J. H. Hitchcock & A. J. Onwuegbuzie (Eds.), The Routledge handbook for advancing integration in mixed methods research. Routledge. https://doi. org/10.4324/9780429432828-16 Brevik, L. M. & Hellekjær, G. O. (2018). Outliers: Upper Secondary School Students Who Read Better in the L2 than in L1. International Journal of Educational Research, 89, 80–91. https://doi. org/10.1016/j.ijer.2017.10.001 Brevik, L. M., & Buchholtz, N. F. (2022). The use of mixed methods to study language learning beyond the classroom. In H. Reinders, C. Lai, & P. Sundqvist (Eds.), The Routledge handbook of language learning and teaching beyond the classroom (pp. 340–353). Routledge. https://doi.org/10.4324/ 9781003048169-28 Brevik, L. M., & Holm, T. (2022). Affinity and the classroom: Informal and formal L2 learning. ELT Journal, 76. https://doi.org/10.1093/elt/ccac012 Brevik, L. M., Reedy, G., Breivik, J., Thue, T., & Barreng, R. (2022, 28 March). Driving pedagogical innovation. Circle U. [Café seminar] University of Oslo. www.uio.no/english/about/news-and-events/ events/circle-u/circle-u-cafe-connecting-researchand-education.html Campbell, L., Gunter, G., & Kenny, R. (2019). Evaluating social change games: Employing the RETAIN model. International Journal of Game-Based Learning, 9(4), 31–44. https://doi.org/10.4018/ IJGBL.2019100103 Clark, D., Tanner-Smith, E., & Killingsworth, S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational Research, 86(1), 79–122. https://doi. org/10.3102/0034654315582065

A “MIXED METHODS WAY OF THINKING” IN GAME-BASED RESEARCH INTEGRATIONS

Cruaud, C. (2018). The playful frame: Gamification in a French-as-a-foreign-language class. Innovation in Language Learning and Teaching, 12(4), 330–343. https://doi.org/10.1080/17501229.2016.1213268 Collins, K., Onwuegbuzie, A., & Sutton, I. (2006). A model incorporating the rationale and purpose for conducting mixed methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4, 67–100. https://eric. ed.gov/?id=EJ797679 Costa, S., Tavares, M., da Silva, B., Isca, B., & Cerol, F. (2021). Hate speech in video games and in online gaming communities – A state of art. Revista Comunicando, 9(1). https://eur-lex.europa. eu/legal-content/EN/TXT/?uri=LEGISSUM:l33178 Darras, J. (2021). The benefits of gaming [Master’s thesis, University of Oslo]. www.duo.uio.no/ handle/10852/92218 Education First. (2022). EF English proficiency index. www.ef.no/epi/ Eriksen, T. M., & Brevik, L. M. (2022). Developing a ‘research literacy way of thinking’ in initial teacher education: Students as co-researchers. In I. Menter (Ed.), The Palgrave handbook of teacher education research (pp. 231–256). Palgrave. https://doi. org/10.1007/978-3-030-59533-3_9-1 Fetters M. D., & Freshwater, D. (2015). The 1 + 1 = 3 integration challenge. Journal of Mixed Methods Research, 9, 115–117. https://doi.org/10.1177/ 1558689815581222 Fetters, M. D., & Molina-Azorin, J. F. (2017). The Journal of Mixed Methods Research starts a new decade: The mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11(3), 291–307. http://doi. org/10.1177/1558689817714066 Gee, J. P. (2017). Teaching, learning, and literacy in our high-risk high-tech world. A framework for becoming human. Teachers College Press. Gorbanev et al. (2018). A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Medical Education Online, 23, 1438718. https://doi.org/ 10.1080/10872981.2018.1438718 Greene, J. C. (2007). Mixed methods in social inquiry. Wiley & Son. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational Evaluation and Policy Analysis, 11, 255–274. https://doi. org/10.3102/01623737011003255 Guetterman, T. C., Molina-Azorin, J. F., & Fetters, M. D. (2020). Virtual special issue on “integration in mixed methods research”. Journal of Mixed Methods Research, 14(4), 430–435. https://doi.org/10.1177/ 1558689820956401 Hitchcock, J. H., & Onwuegbuzie, A. (2022). The Routledge handbook for advancing integration in mixed methods research. Routledge.

359

Ho, W. (2019). ‘I knew that you were there, so I was talking to you’: The use of screen-recording videos in online language learning research. Qualitative Research. https://doi.org/10.1177/14687941198 85044 Internet World Statistics. (2022). Usage and population statistics. www.internetworldstatistics.com/ stats.htm Johnson, R. B. (2017). Dialectical pluralism: A metaparadigm whose time has come. Journal of Mixed Methods Research, 11, 156–173. https://doi. org/10.1177/1558689815607692 Mertens, D. M. (2022). Mixed methods integration for transformative purposes. In J. H. Hitchcock, & A. Onwuegbuzie (Eds.), The Routledge handbook for advancing integration in mixed methods research (pp. 71–85). Routledge. https://doi. org/10.4324/9780429432828-7 NESH. (2019). Guidelines for research ethics in the social sciences, humanities, law and theology. www.forskningsetikk.no/en/guidelines/socialsciences-humanities-law-and-theology/guidelinesfor-research-ethics-in-the-social-sciences-humanitieslaw-and-theology/ Onwuegbuzie, A. J., Hitchcock, J., Natesan, P., & Newman, I. (2018). Using fully integrated Bayesian thinking to address the 1+1=1 integration challenge. International Journal of Multiple Research Approaches, 10, 1–13. https://doi.org/10.29034/ IJMRA.V10N1A43 Poth, C. N. (2018). Innovation in mixed methods research. A practical guide to thinking with complexity. Sage. Rüth, M. & Kaspar, K. (2021). Commercial video games in school teaching: Two mixed methods case studies on students’ reflection processes. Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg. 2020.594013 Scholtz, K. (2022). Digital game-based language learning in extramural settings. In H. Reinders, C. Lai, & P. Sundqvist (Eds.), The Routledge handbook of language learning and teaching beyond the classroom (pp. 129–141). Routledge. https://doi. org/10.4324/9781003048169-12 Schoonenboom, J. (2022). Developing the metainference in mixed methods research through successive integration of claims. In J. H. Hitchcock, & A. Onwuegbuzie (Eds.), The Routledge handbook for advancing integration in mixed methods research (pp. 55–70). Routledge. https://doi.org/10.4324/9780429432828-6 Schuster-Amft, C., Eng, K., Lehmann, I., Schmid, L., Kobashi, N., Thaler, I., Verra, M., Henneke, A., Signer, S., McCaskey, M., & Kiper, D. (2014). Using mixed methods to evaluate efficacy and user expectations of a virtual reality-based training system for upper-limb recovery in patients after stroke: A study protocol for a randomised

360

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

controlled trial. Trials, 15(350). https://doi.org/ 10.1186/1745-6215-15-350 Schwarz, M. (2020). Beyond the walls: A mixed methods study of teenagers’ extramural English practices and their vocabulary knowledge [Doctoral dissertation, University of Vienna]. https:// doi.org/10.25365/thesis.63632 Shannon-Baker, P. (2022). Virtual special issue on “Mixed methods designs, integration, and visual practices in educational research.” Journal of Mixed Methods Research, 16(2), 159–164. https:// doi.org/10.1177/15586898221083959 Sieber, J., & Tolich, M. (2013). Planning ethically responsible research. Sage. https://doi.org/10.4135/ 9781506335162

Sletten, M., Strandbu, Å., & Gilje, Ø. (2015). Idrett, dataspilling og skole –konkurrerende eller «på lag”? [Sporting, gaming and school – competing or complementary?]. Norsk Pedagogisk Tidsskrift, 5, 334–350. https://doi.org/10.18261/ISSN15042987-2015-05-03 Tashakkori, A., Johnson, R. B., & Teddlie, C. (2020). Foundations of mixed methods research. Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage. Warland, A. (2018). The feasibility and acceptability of virtual reality gaming technologies for upper-limb stroke rehabilitation: A mixed-methods study. [Doctoral dissertation, Brunel University London]. Retrieved from: http://bura.brunel.ac.uk/handle/2438/19031

25 Integrating Secondary Data from Ethnically and Racially Minoritized Groups in Mixed Methods Research Daphne C. Watkins and Natasha C. Johnson

INTRODUCTION The growing interest in mixed methods research by scholars from disciplines like education, nursing, medicine, the humanities and the social sciences is fuelled by the current social and political climate and the need to address the impact of structural racism on ethnically and racially minoritized (ERM) populations (DeCuir-Gunby, 2020; Watkins, 2022; Watkins & Gioia, 2015). This surge means that more researchers are seeking ways to employ innovative integration approaches in their mixed methods designs. It also underscores the need for more guidance on integrating different data sources to address specific research questions, particularly those addressing social justice, equity and power structures (Watkins, 2022). For additional discussion, see Chapter 13 (this volume). In recent years, some of the guidance on organizing, sharing and integrating data has come from federal agencies, whose newly employed policies now require researchers to outline plans for data sharing (e.g., National Institutes of Health, 2021). Between the uptick in robust data-sharing policies and the growing interest in mixed methods by social justice scholars across various disciplines, more data on ERM groups will become publicly

available for secondary use in the coming years. These “secondary” data include everything from large federally owned data sources to smaller files. Though guidelines for using secondary data are growing in the literature (Beck, 2019; Largan & Morris, 2019; Logan, 2020; Watkins, 2022), resources that outline how to use secondary data from ERM groups are limited. Furthermore, guidance for how to integrate secondary data from ERM groups into mixed methods research is practically non-existent. In this chapter, we discuss how to integrate secondary data from ERM groups into mixed methods research to address challenges related to the underrepresentation of ERM groups in research. We begin by acknowledging Jessica T. DeCuir-Gunby’s (2018, 2020) Critical Race Mixed Methodology (CRMM) and encouraging researchers to use secondary data from ERM groups in CRMM. We then examine the challenges experienced by researchers when dealing with data from ERM groups and how integrating secondary data with a critical race theory lens can be used to address these challenges. We end the chapter with implications for real-world impact discussing future directions for integrating secondary data into mixed methods research using a critical race lens.

362

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

SECONDARY DATA IN MIXED METHODS With the expansion of technology, data collection, data management and analytical procedures have improved. Now more than ever, research teams are exploring more efficient ways to collect, manage and analyze their quantitative and qualitative data (Watkins & Gioia, 2015). These advancements benefit single-method quantitative and qualitative studies, and mixed methods projects. Efficiency in data collection, management and analysis also maximizes existing data for secondary purposes. These secondary data have been collected previously—by you or someone else—and can be used for a research project to address a different or new research question (Watkins, 2022). Secondary data can include quantitative, qualitative or mixed methods files. They can range from large, publicly available data (e.g., US Census data) to smaller, privately-owned data (e.g., local school district data). The movement towards increasing access to secondary data, whether they are case studies, “big data”, papers, artefacts or electronic documents, and the current sociopolitical climate put social justice scholars at a critical moment in time where we can be more intentional about maximizing quantitative, qualitative and mixed methods data for the common good. For the purposes of this chapter, we affirm that using secondary data is not necessarily an innovation that advances mixed methods research. Rather, given previous researchers’ challenges with acquiring, analyzing and disseminating research with and in service of ERM groups, we underscore the value of secondary data from ERM groups—interpreted within their unique racial, ethnic and cultural contexts as an innovation that advances mixed methods research.

SECONDARY DATA IN MIXED METHODS WITH ERM GROUPS Racial disparities in the United States resulting from structural and systemic discriminatory practices have been well documented in the literature (Castle et  al., 2019). Social sciences and health disciplines have only recently begun to acknowledge publicly and apologize for their contribution to systemic racism, broadly and within their respective fields (e.g., American Psychological Association, 2021). Researchers are faced with the reality of how scientific enquiry has been instrumental in the deficit and disparity-inducing research approaches that are still present today.

For example, early scholars were biased when working with ERM groups in single-method (Marshburn et  al., 2021), and mixed and multimethod studies (Shannon-Baker, 2021). When members from ERM groups were included in research studies, their experiences were frequently treated as problematic and viewed from a deficit perspective (Syed et al., 2018). Social justice scholars’ growing interest and tenacity are shaping how the next generation of mixed methods researchers (Mertens, 2012, 2013; Watkins, 2022) manage secondary data in mixed methods research with ERM groups. More ERM scholars are challenging the status quo of what was historically considered “gold standard” research—which is often defined using Western, predominately White samples—and instead applying a more culturally rich lens to the work they do with and in service of ERM groups (Adams et al., 2015). One way to do this is to apply Critical Race Theory to mixed methods research designs, which helps focus the research questions on aspects of race, racism and power (DeCuir-Gunby, 2018). Inspired by DeCuir-Gunby’s previous work on Critical Race Mixed Methodologies, we underscore its value and invite scholars to use Critical Race Theory (CRT) in mixed methods with secondary ERM samples.

SECONDARY DATA IN CRITICAL RACE MIXED METHODOLOGY WITH ERM GROUPS CRT challenges racism and power structures (Bell, 1993). CRT was originally designed to challenge the creation and maintenance of White supremacy and the subordination of people of colour in the legal system (Bell, 1995; Crenshaw et al., 1995). In her pinnacle 2018 paper, DeCuirGunby (2018) outlined ten key CRT principles as the centrality of race and racism (Bell, 1993), challenge to dominant ideology (Bell, 1995), the property rights of whiteness (Harris, 1994), intersectionality (Crenshaw, 1989), the myth of meritocracy (Gotanda, 1991), the centrality of experiential knowledge (Delgado & Stefancic, 2017), the historical/contextual perspective (Delgado & Stefancic, 2017), commitment to social justice (Peller, 1990), interdisciplinarity (Chang, 1993) and the reinterpretation of civil rights outcomes/ interest convergence (Bell, 1980). CRMM frame traditional mixed methods studies using a CRT lens. CRMM is an innovative approach to integrating qualitative and quantitative data because, as DeCuir-Gunby and Schutz

INTEGRATING SECONDARY DATA FROM ETHNICALLY AND RACIALLY MINORITIZED GROUPS 363

(2018) argue, studies in critical race theory are often qualitatively driven, and mixed methods studies are frequently quantitatively driven. Thus, combining CRT and mixed methods is a natural progression in the study of racism and power structures, and a way to advance the science of mixed methods research. Furthermore, CRMM differs from traditional mixed methods approaches in three fundamental ways. First, CRMM is grounded in critical race theory, and not all mixed methods studies are grounded in theory (Greene, 2007). Second, CRMM focuses on race, which is not always underscored in mixed methods research (DeCuir-Gunby, 2020; Watkins, 2022). Finally, the larger goal for CRMM is to critique power structure in a way that results in social justice and real-world change (DeCuir-Gunby, 2020). Despite long-standing research on transformative designs, social justice is not always a requirement for mixed methods studies (Mertens, 2012, 2013). Just as DeCuir-Gunby and colleagues (2018) and DeCuir Gunby (2020) suggested combining CRT and mixed methods (e.g., CRMM), we encourage scholars to take this a step further and find ways to integrate secondary data sources from ERM populations into their mixed methods studies. This line of thinking is relatively new and has not been explicitly noted in the literature. Therefore, scholars who want to address racism and power structures in their work should integrate the core principles of CRMM (DeCuirGunby et  al., 2018; DeCuir-Gunby, 2020) with the evaluative principles for integrating secondary data (Watkins, 2022) into mixed methods with ERM samples. Qualitative research has dominated previous CRT studies, and quantitative data are the more commonly used secondary data source (Smith, 2008; Watkins, 2022) in mixed methods

Critical Race Mixed Methodology

research. Combining CRMM (DeCuir-Gunby, 2020; DeCuir-Gunby & Schutz, 2018) and secondary data in mixed methods research (Watkins, 2022) allows us to apply a CRMM+SD approach to mixed methods studies that integrate secondary data from ERM samples (Figure 25.1). The CRMM+SD approach to mixed methods is inspired by previous work from CRT mixed methods scholars. It also acknowledges previous mixed methods scholars who have advocated for transformative approaches to addressing social justice and combating injustice (Andrzejewski et  al., 2019; Mertens, 2012, 2013). However, one clear distinction between transformative mixed methods and the CRMM+SD approach is that the latter names CRT and its principles as the lens through which the secondary data are evaluated and integrated into the mixed methods study. This integration approach elevates the contextual experiences of research done with and in service of members of ERM groups.

INTEGRATING SECONDARY DATA FROM ERM GROUPS INTO MIXED METHODS: CHALLENGES AND SOLUTIONS When applying a CRMM+SD approach to mixed methods studies with ERM samples, we acknowledge previous researchers’ challenges and see them as opportunities to advance research in these areas of study. For example, previous researchers have reported challenges when recruiting and retaining ERM participants in research (RivasDrake et al., 2016). Despite the challenges faced when working with ERM samples, there are feasible solutions. Yet, poor decisions made by

Critical Race Mixed Methodology with Secondary Data (CRMM+SD)

Secondary Data

Figure 25.1  Critical race mixed methodology with secondary data (CRMM+SD) Source: Author created.

364

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 25.1  Common challenges and solutions when integrating secondary data from ERM groups in mixed methods research Challenge

Definition

Integration solution

Small sample sizes

Low numbers of members from ERM groups in the quantitative sample.

Researcher bias

Researcher positionality, training, experience and skill set that influence how research is done; might limit options and creativity when working with members of ERM groups. Limited financial resources for research with ERM groups.

Supplement SD from ERM groups with new data collection (or vice versa) to fill the gaps resulting from the limitations of the SD. Use SD (or collect new data) from ERM groups to address limitations associated with researcher bias.

Lack of funding

Diversify funding sources and emphasize the strength of using SD in MMR with ERM groups.

Note: ERM = ethnically and racially minoritized MMR = mixed methods research SD = secondary data Source: Author created.

researchers who sample from ERM groups led to misinterpretations of the study’s outcomes and inaccurate results (Datta, 2018; Tao, 2021). This is even more problematic when doing mixed methods research with ERM groups. These poor decisions often centre around three challenges: small samples, researcher bias and a lack of funding (Table 25.1). Below, we describe each challenge and how researchers can address the challenges and elevate the needs of ERM groups in their mixed methods research using new or secondary data with a critical race theory lens.

INTEGRATION CHALLENGES WITH SMALL SAMPLES OF ERM GROUPS Scholars who focus on the lived experiences of ERM groups have made incredible strides to build, sustain and disseminate theoretical knowledge and empirical evidence in service to these groups (Crenshaw, 1991; Solorzano & Bernal, 2001; Williams et al., 1997). Unfortunately, ERM groups remain understudied compared to predominately White groups. A common challenge observed with research studies with diverse samples is that the ERM groups are under-sampled, sometimes resulting in severely low sample sizes that lack sufficient power to execute appropriate statistical analyses. The under-sampling of ERM groups has resulted in researchers collapsing these under-sampled participants into “other” ethnic/racial categories. Research has shown that the aggregating of ERM data limits the

interpretation of results and documented nuance of their lived experiences (Lett et  al., 2022). Another issue related to under-sampling is the difficulty with publishing studies with low sample sizes. The privileging of large samples and statistically significant findings in quantitative research further limits the dissemination and investigation of the experiences of people from ERM groups (Datta, 2018). By way of example, let’s consider Chelle’s dilemma to examine the issue of small ERM sample sizes. Chelle is a doctoral student studying Black Caribbean adolescents’ support systems and help-seeking behaviours. Chelle is lucky because a publicly available data source includes their constructs of interest and a large sample of Black/African American adolescents. Unfortunately, the original researchers did not ask Black/African American respondents to define their heritage further. So, Chelle cannot identify which of the participants are of Caribbean descent. Chelle knows that Black adolescents of Caribbean descent are understudied, but does not think they have the resources to conduct a large-scale study needed to answer their research question. What should Chelle do?

Sampling Solutions The challenge of under-sampling ERM groups can be addressed using secondary data in mixed methods research. While the under-sampling of ERM groups can be frustrating to researchers, there are at least two ways to address this

INTEGRATING SECONDARY DATA FROM ETHNICALLY AND RACIALLY MINORITIZED GROUPS 365

challenge. Given the nature of qualitative data, smaller sample sizes are rarely an issue. So the first way that researchers can address the challenge of small samples due to under-sampling ERM groups involves doing something about the quantitative data. Depending on the research question, scope and study budget, researchers interested in doing mixed methods with a small quantitative sample of ERM should consider collecting new—or evaluating secondary (Watkins, 2022) qualitative data to expound on the information gleaned from the small quantitative sample. In the case of Chelle, because they cannot discern which participants from their quantitative data are of Caribbean descent, it makes sense for them to conduct a qualitative study. The qualitative study can build on the secondary quantitative study and help Chelle further examine Caribbean adolescents’ unique characteristics and experiences. Chelle has several options. One option is to reach out to the participants from the original study. If they know the original researchers of the quantitative data source, they can reach out to them to see if it is possible to track down the original study participants for a new, qualitative follow-up study that would allow Chelle to probe more into participants’ ethnic and racial heritage. Another option is for Chelle to recruit new participants for a new qualitative study. They can build the interview protocol based on the findings from the public data, which will help Chelle interpret the secondary study’s conclusions. Essentially, the qualitative study component can strengthen the small sample of ERM participants by addressing questions that add racial and ethnic context to the secondary quantitative dataset. A second way to address the challenge of small sample sizes using secondary data in mixed methods is to combine data from the first secondary quantitative dataset with ERM data from another secondary (or primary) quantitative data source. Ideally, it would be best if the demographic characteristics of the respondents from the quantitative data were the same or similar across datasets. Combining two or more datasets whose participant characteristics and variables are aligned can increase the total sample size (i.e., the “n”) and generate the power needed for appropriate statistical analyses. Overcoming the challenge of a small sample size is easy when researchers use a primary or secondary data source that complements the other data source. Using qualitative data can provide context that enhances the interpretability of results and combats the statistical power challenges resulting from small quantitative sample sizes.

INTEGRATION CHALLENGES WITH RESEARCHER BIAS WITH DATA FROM ERM GROUPS Regardless of training and experience, all researchers have biases regarding certain aspects of their work. This is not necessarily a negative characteristic; it is just the nature of research. Many of us are trained in particular ways, and how we execute our work is a direct byproduct of our positionality, training, experiences and skillsets. Therefore, we often do not consider other methodologies, methods or ways to see our work if it falls outside our training and area of expertise. An example is when a researcher is qualitatively oriented and decides to address a quantitative question (e.g., the prevalence of a condition) using a qualitative method (e.g., a focus group). Although the focus group might help illuminate some aspects of the research question, if the nature of the question is post-positivist and, therefore, quantitative, the researcher should use a quantitative method, regardless of their training. Another example is when a researcher whose research typically focuses on adults decides to extend their research to adolescents’ experiences but does not appropriately adapt their survey design or recruitment strategy to address the unique needs of this population. The lack of using developmentally appropriate questionnaires can result in unreliable data. Unfortunately, the consequences of our biases are seen at later stages of the research process, which is difficult to correct retroactively. Dan is a media studies scholar investigating Latinx college students’ experiences with ethnic-racial discrimination. Dan’s colleague agreed to let him use interview data from a recently completed project focused on first-year college students’ experiences. Even though the interview protocol did not explicitly include questions about discrimination, Latinx students discussed their experiences with ethnic/ racial discrimination in the interviews. After reviewing the data, Dan realized a considerable flaw in the interview methods. Respondents who discussed discrimination often discussed being mistreated because of their ethnicity, but the interviewers did not probe further to identify the types of unfair treatment students experienced (e.g., verbal, physical, denied opportunity). What does Dan do now? How can he address this missed opportunity in the primary researcher’s interview procedures now that the data have been collected?

Minimizing Bias Solutions Unfortunately, biases exist and can make themselves evident during a research project’s design,

366

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

data collection and data recruitment stages, ultimately creating challenges for researchers who wish to use the previously collected data for new research questions (e.g., secondary data). Furthermore, bias can persist when using ERM samples as secondary data in mixed methods research, negatively affecting how the research is analyzed and interpreted. If you are the researcher with access to the data set in which there is apparent bias in the methods, then decisions will need to be made about using a different data source to supplement the bias in the secondary data. Let’s revisit the example with Dan from above. Although he is interested in Latinx students’ discriminatory experiences, the primary researchers did not adequately probe respondents when they discussed discriminatory experiences. This was a missed opportunity to unpack the types of discriminatory experiences Latinx students reported, but all is not lost. Dan can use the findings from the qualitative interviews and the conclusions of that study to inform the next steps in his research (e.g., exploratory sequential mixed methods design). In a follow-up quantitative phase of his mixed methods study, Dan can recruit a sample of Latinx students to complete an online questionnaire that will probe further into the types of ethnic/racial discrimination students experience. In Dan’s case, the important takeaway is for him to realize that even though secondary data may have limited use due to the bias within the primary data collection methods, there are opportunities for him to build on those data. He can integrate previous data with new or secondary quantitative data to further explore his research topic within a mixed methods research design.

INTEGRATION CHALLENGES WHEN FUNDING RESEARCH WITH ERM GROUPS While there are challenges at the stages of participant recruitment (e.g., low sample sizes) and data collection (e.g., limited use of data due to researcher bias), we would be remiss if we did not discuss systemic level challenges with research that focuses on the lived experiences of ERM groups. For example, funding availability varies by country and across geographic regions. Even with the recent surge in US federal and private funding aimed at addressing the consequential impacts of racism across disciplines (Centers for Disease Control and Prevention [CDC], 2021; Ford Foundation, 2020; White House, 2021), scholars who have longstanding documented histories of doing culturally relevant

research with ERM groups are still combatting the challenges associated with lack of funding. Specifically, in recent times, although funding sources have increased to address ERM needs, funding disparities still exist because researchers with little to no experience of working with and in service of ERM groups are receiving funding, furthering the funding disparities within research that focus on ERM groups (McFarling, 2021). Historical and contemporary issues with funding create challenges with the scalability, robustness and execution of studies with minoritized groups. Access to and availability of funding is essential to do the deep and necessary work with ERM populations. Let’s turn our attention to an early career scholar, Jade, who has a developing research programme focusing on violence against indigenous women, and strategies supporting their safety and mental wellness. Jade’s past research has been quantitative, but emerging research requires mixed methodologies to address her research questions adequately. Jade is concerned about the likelihood of being awarded and carrying out a mixed methods project, especially because she has not received formal training in mixed methods research. She has been advised to apply for a prestigious federal grant because it would look favourable for her pre-tenure review. More importantly, she will have the resources to ask questions she has not been able to answer with her current data. Jade is torn about what to do.

Funding Solutions Regardless of the topic, most researchers will say they experience challenges securing funding for their work. Despite this, data suggests that researchers from ERM backgrounds experience more challenges when securing funding than their White counterparts (Taffe & Gilpin, 2021). While there is no perfect formula to guarantee funding, researchers can be strategic in ensuring the continuation of their research programme through various methods. In the case of Jade (above), there are tactical strategies that she can use to attract potential funding when doing mixed methods with secondary data from ERM groups. One approach is for Jade to pay close attention to her research programme and base her research project on her needs at this stage rather than on the most popular or prestigious funding source. For example, early-career researchers like Jade might benefit from small, internal funding sources to fund their pilot and exploratory research. Once the findings from these early studies are published, Jade can seek funding

INTEGRATING SECONDARY DATA FROM ETHNICALLY AND RACIALLY MINORITIZED GROUPS 367

from regional agencies, foundations or professional associations and organizations to integrate new and secondary data sources through mixed methods research with ERM groups. Once these projects’ results are published, Jade can seek additional funding from national organizations, sources and foundations. Also noteworthy is the importance of diversifying funding sources when working with ERM groups. Some potential funding sources will have different goals than others and allow for more nontraditional and culturally relevant research methods. For example, because foundations often want to get to know the applicants and work with them on their applications, there are often opportunities to discuss culturally relevant sampling strategies with foundation representatives before applying for funding. This contrasts with federal agencies that frequently share a funding announcement with detailed instructions and accept applications during certain times of the year. Given the nature of the research with ERM groups and the timeline for the work, one funding source might be more appropriate at a specific time than another. Another strategy Jade can use to continue her research programme is to be intentional about how she “pitches” her integration of secondary data in her new mixed methods project within her grant applications. For example, when integrating secondary data from ERM samples into a mixed methods project, the researcher maximizes the resources invested into the secondary data source and stewards the resources requested from a new funding agency. In short, when using secondary data, researchers often require less funding than when doing new, primary data collection. This can save money, time and people power. This might be attractive to potential funders and demonstrate intentionality that results in the requested financial support.

IMPLICATIONS AND FUTURE DIRECTIONS Mixed methods research can be used to examine social and economic inequities faced by individuals from ERM groups. Unfortunately, we will not benefit from mixed methods research with ERM groups if we do not address the challenges that scholars face when sampling, managing and integrating data from these underrepresented groups. One way to address these challenges is to integrate secondary data into mixed methods research with ERM groups using a critical race theory lens (i.e., CRMM+SD). There are innumerable advantages to doing mixed methods research with ERM groups. Still, small samples, researcher bias, and

lack of funding prevent researchers from maximizing the integration of secondary data from ERM groups into their mixed methods studies. This chapter acknowledges these challenges and offers solutions for integrating secondary data into mixed methods research using a critical race theory lens. In addition to the methodological decisions to address the small sample sizes described in this chapter, future researchers should also address the challenge of ERM groups being understudied and under-sampled by increasing education and training opportunities that focus on advanced research methods to address the needs of ERM groups broadly. ERM students often pursue higher education to learn more about their communities (Wang et  al., 2021) and develop strategies to improve their communities’ health, social and economic conditions. Centring the interests of ERM students—and creating targeted opportunities to elevate these interests through a CRT lens—will address challenges associated with ERM groups being underrepresented in research. Further, it is necessary to tailor training programmes that have historically been based on Western research methods (Datta, 2018; Hayward et  al., 2021; Lewis, 2016) to include culturally appropriate methods for research and integration that elevate the experiences of ERM groups (e.g., community-based participatory research). Dealing with bias and apparent gaps in secondary data with ERM groups can deter researchers from doing mixed methods research. But there are solutions and workarounds for most scenarios. In addition to the solutions outlined in this chapter, future researchers should stretch beyond their comfort zones when working with secondary data from ERM groups and collecting primary data from ERM samples. For example, perhaps when recruiting for a new project with American Native youth in the community, you learn that Native youth frequent coffee shops and malls. In previous studies with Latinx youth, you recruited them from the skating rink, the gym and the local community centre. These locations are also where your community liaisons frequent. Just because this is how you have recruited other ERM youth for previous research projects, does not mean this recruitment strategy will work for your new study with Native youth. Going with your original plan could result in a missed opportunity. Therefore, it is important for researchers to acknowledge their bias and, regardless of their experience, seek advice on the best way to proceed with sampling and integrating data from ERM groups on a case by case basis. If researchers working with ERM groups do not acknowledge their bias and take action to combat missteps and stereotypes, they will face

368

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

challenges that could have a long-term impact on their research and the field. For further discussions of prioritizing cultural responsiveness in mixed methods research with underrepresented communities, see Chapter 16 (this volume). Addressing this challenge involves taking appropriate steps, such as creating culturally relevant recruitment techniques, including ERM team members in all research projects, and referring to recent literature on non-traditional research methods used with ERM populations. Furthermore, another best practice is maintaining an anti-racist (e.g., critical race theory) lens when collecting and integrating data from ERM groups. This mindset holds us accountable and increases our chances of applying culturally appropriate methods to the communities that need our best work. Researchers must be intentional about the types of funding sought for studies where they intend to integrate secondary data from ERM groups into mixed methods research. Decisions around funding are vital to a successful research career. But we would be remiss to overlook the importance of advocacy and using one’s voice to address the lack of funding for research with ERM groups (Lazdins, 2021). A significant role in addressing equity and combatting systemic racism and discrimination in funding inequities is for all researchers—not just those from ERM communities— to advocate for more resources (i.e., funding, technical assistance, consultation, etc.) for research that involves ERM groups. Advocating for more resources will yield greater benefits, as it not only helps to identify funding to produce high-quality research with and in service of ERM groups, but it also addresses the lack of funding for research with ERM groups on behalf of future researchers. In short, while building their research portfolio that integrates secondary data from ERM groups using a CRT lens, scholars can also advocate for more resources for themselves and future generations of scholars (Majic & Strolovitch, 2020).

CONCLUSION This chapter discusses integrating secondary data from ethnically and racially marginalized (ERM) groups into mixed methods research. Integrating secondary data in research central to ERM communities using a critical race theory (CRT) lens addresses the challenge of low sample size, researcher bias and can inform the lack of funding allocated for research with ERM groups. Similarly, if we promote culturally appropriate research methods (Tao, 2021) that answer research

questions but run counter to the “gold standard” we attach to traditional western research methods (e.g., power with large samples), we can work in service of ERM groups to elevate their experiences and make sure their voices are heard (Watkins, 2022). The integration approaches outlined in this chapter will springboard future research that uses secondary data and advance both single-method and mixed methods research with ERM groups using CRT.

WHAT TO READ NEXT DeCuir-Gunby, J. T. (2020). Using critical race mixed methodology to explore the experiences of African Americans in education. Educational Psychologist, 55(4), 244–255.

In this paper, DeCuir-Gunby explores the relationship between her positionality, personal experiences and enquiry worldview, and how they informed the development of Critical Race Mixed Methodology (CRMM). DeCuir-Gunby begins by describing educational psychology, race and racialized experiences that occurred to her throughout her life. She then explores her enquiry worldview, critical race theory, its origins, fundamental principles and the ways she has been able to apply it to her research on science, technology, engineering and mathematics (STEM) with Black girls. Audiences should read this paper because it provides a strong foundation for Critical Race Theory, a detailed account of the origins of Critical Race Mixed Methodology, and how DeCuir-Gunby has been able to successfully apply CRMM to her quantitative and qualitative data from Black girls in STEM to push the field forward. Watkins, D.C. & Gioia, D. (2015). Mixed methods research. Oxford University Press.

Written with social work professionals in mind, this practical guide to the mixed methods paradigm illustrates how to effectively handle the integration of different philosophical and empirical approaches to research. Watkins and Gioia review the fundamentals of mixed methods research designs and the general suppositions of mixed methods procedures, look critically at mixed method studies and models that have already been employed in social work, and reflect on the contributions of this work to the field. Audiences should read this pocket guide because the authors describe six major types of mixed methods designs, present data collection and

INTEGRATING SECONDARY DATA FROM ETHNICALLY AND RACIALLY MINORITIZED GROUPS 369

analysis fundamentals, and flag the perils and pitfalls of undertaking mixed methods. Watkins, D. C. & Johnson, N. C. (2023). Advancing education research through mixed methods with existing data. In Ivankova N. and Guetterman T. (Ed). International Encyclopedia of Education (4th ed.). Elsevier.

This chapter reviews mixed methods with existing qualitative and quantitative data. It discusses using the knowledge-level continuum as a framework for mixed methods with existing data and strategies for making sense of available information in education research. Audiences should read this chapter because the authors examine how using existing data adds to the unique possibilities for the future of education research. Previous studies suggest existing data are frequently used at the beginning (i.e., for the first single-method phase) of a mixed methods study. However, as qualitative and quantitative data become more robust, the potential for implementing mixed methods studies with existing data in education research is more realistic. Watkins, D.C. (2022). Secondary data in mixed methods research. Sage.

In this book, Watkins discusses how using secondary (qualitative and quantitative) datasets in mixed methods research can help answer new and ongoing research questions. The text describes strategies for sifting through data records and making sense of the available information associated with secondary data. Increasing our use of secondary data sources adds to the unique possibilities for further defining and operationalizing mixed methods research. Previous studies suggest secondary data are usually used initially (or for the first single-method phase) of a mixed methods study (Gray & Geraghty, 2020; Smith, 2008). Audiences should read this book because as qualitative and quantitative data sources become more robust and sophisticated, there will be more opportunities to implement both components of mixed methods studies using previously collected data for purposes aligned (as well as those not aligned) with the original research.

REFERENCES Adams, G., Dobles, I., Gómez, L. H., Kurtiş, T., & Molina, L. E. (2015). Decolonizing psychological science: Introduction to the special thematic section. Journal of Social and Political Psychology, 3(1), 213–238. https://doi.org/10.5964/jspp.v3i1.564 Andrzejewski, C. E., Arnberg, B., & Baggett, H. C. (2019). Transformative mixed methods: A missed

opportunity. Research Methods for Social Justice and Equity in Education, 241–251. https://doi.org/ 10.1007/978-3-030-05900-2_20 American Psychological Association. (October, 2021). Apology to people of color for APA’s role in promoting, perpetuating, and failing to challenge racism, racial discrimination, and human hierarchy in U.S. www.apa.org/about/policy/racism-apology Beck, C.T. (2019). Secondary qualitative data analysis in the health and social sciences. Routledge Bell, D. A. (1980). Brown v. Board of Education and the Interest-Convergence Dilemma. Harvard Law Review, 93(3), 518–533. https://doi.org/10.2307/ 1340546 Bell, D. A. (1993). Faces at the bottom of the well: The permanence of racism. Basic Books. Bell, D. A. (1995). Who’s afraid of critical race theory. University of Illinois Law Review, 4, 893–920. Burlew, A. K., Peteet, B. J., McCuistian, C., & MillerRoenigk, B. D. (2019). Best practices for researching diverse groups. American Journal of Orthopsychiatry, 89(3), 354–368. https://doi.org/10.1037/ort00 00350 Castle, B., Wendel, M., Kerr, J., Brooms, D., & Rollins, A. (2019). Public Health’s Approach to Systemic Racism: A Systematic Literature Review. Journal of Racial and Ethnic Health Disparities, 6, 27–36. https://doi.org/10.1007/s40615-0180494-x Centers for Disease Control and Prevention (CDC). (2021). CDC’s Efforts to Address Racism as a Fundamental Driver of Health Disparities. www.cdc. gov/healthequity/racism-disparities/cdc-efforts. html Chang, R. S. (1993). Toward an Asian American legal scholarship: Critical race theory, poststructuralism, and narrative space. California Law Review, 81(5), 1241–1323. https://doi.org/10.2307/3480919 Chapman, T. K., Hartlep, N. D., Vang, M., LipseyBrown, T., & Joseph, T. (2019). Not one, but many: A CRT research team approach to investigate student experiences in racially diverse settings. In J. T. DeCuir-Gunby, T. K. Chapman, & P. A. Schutz (Eds.), Understanding critical race research methods and methodologies: Lessons from the field (pp. 86–96). Routledge. Crenshaw, K. (1989). Demarginalizing the intersection of race and sex: a Black feminist critique of antidiscrimination doctrine, feminist theory, and antiracist politics. University of Chicago Legal Forum, 1989, 139–167. Crenshaw, K. (1991). Mapping the margins: intersectionality, identity politics, and violence against women of color. Stanford Law Review, 43(6), 1241–1299. https://doi.org/10.2307/1229039 Crenshaw, K., ed. (1995). Critical Race Theory: The Key Writings that Formed the Movement. New York, NY: The New Press.

370

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Datta, R. (2018). Decolonizing methodologies: a transformation from science-oriented researcher to relational/participant-oriented researcher. American Culture and Research Journal, 42(1): 115–130. https://doi.org/10.17953/aicrj.42.1.datta DeCuir-Gunby, J. T. (2020). Using critical race mixed methodology to explore the experiences of African Americans in education. Educational Psychologist, 55(4), 244–255. DeCuir-Gunby, J. T., Chapman, T. K., & Schutz, P. A. (Eds.). (2018). Understanding critical race research methods and methodologies: Lessons from the field (1st ed.). Routledge. https://doi.org/10.4324/ 9781315100944 DeCuir-Gunby, J. T., & Schutz, P. A. (2018). Critical race mixed methodology: Designing a research study combining critical race theory and mixed methods research. In Understanding critical race research methods and methodologies (pp. 166– 179). Routledge. Delgado, R., & Stefancic, J. (2017). Critical race theory: an introduction (3rd ed.). New York University Press. Fetters, M. D. & Freshwater, D. (2015). The 1 + 1 = 3 integration challenge. Journal of Mixed Methods Research, 9(2), 115–117. https://doi.org/10.1177/ 1558689815581222 Ford Foundation (2020). Ford Foundation Announces $180 Million in New Funding for U.S. Racial Justice Efforts. October 10. www.fordfoundation.org/ the-latest/news/ford-foundation-announces-180million-in-new-funding-for-us-racial-justice-efforts/ Gray, J. & Geraghty, R. (2020). Using quantitative data in qualitative secondary analysis. In K. Hughes, and A. Tarrant (eds.) Qualitative Secondary Analysis. 1st ed, 3–18. London, UK: Sage Publications. Greene, J. C. (2007). Mixed methods in social inquiry (Vol. 9). John Wiley & Sons. Gotanda, N. (1991). A critique of “Our constitution is color-blind.” Stanford Law Review, 44(1), 1–68. https://doi.org/10.2307/1228940 Harris, A. P. (1994). The jurisprudence of reconstruction. California Law Review, 82(4), 741–785. https://doi.org/10.2307/3480931 Hayward, A., Wodtke, L., Craft, A., Robin, T., Smylie, J., McConkey, S., Nychuk, A., Healy, C., Star, L., & Cidro, J. (2021). Addressing the need for indigenous and decolonized quantitative research methods in Canada, SSM – Population Health, 15, 100899, https://doi.org/10.1016/j.ssmph.2021.100899 Largan, C., & Morris, T. (2019). Qualitative secondary research: A step-by-step guide. Sage. Lazdins, E. (2021). Help advocate for increased IDEA, EHDI, and research funding. July 16, 2021. https:// leader.pubs.asha.org/do/10.1044/2021-0716-ideaehdi-funding/full/?utm_source=hearingtracker. com

Lett, E., Asabor, E., Beltrán, S., Michelle Cannon, A., & Arah, O. A. (2022). Conceptualizing, contextualizing, and operationalizing race in quantitative health sciences research. Annals of Family Medicine, 2792. Advance online publication. https:// doi.org/10.1370/afm.2792 Lewis, C. (2016). Understanding research methods to study African American males in college. Journal of Negro Education, 85(1): 3–15. http://dx.doi. org/10.7709/jnegroeducation.85.1.0003 Logan, T. (2020). A practical, iterative framework for secondary data analysis in educational research. Australian Educational Researcher, 47, 129–148. https://doi.org/10.1007/s13384-019-00329-z Majic, S., & Strolovitch, D. (2020). Editors’ introduction: Mentoring and marginalization. PS: Political Science & Politics, 53(4), 763–769. https://doi. org/10.1017/S1049096520000773 Marshburn, C.K., Folberg, A. M., Crittle, C., Maddox, K. B. (2021). Racial bias confrontation in the United States: What (if anything) has changed in the COVID-19 era, and where do we go from here? Group Processes & Intergroup Relations, 24(2): 260–269. https://doi.org/10.1177/136843 0220981417 McFarling, U. L. (2021, September). ‘Health equity tourists’: How white scholars are colonizing research on health disparities. STAT News. www. s t a t n e w s . c o m / 2 0 2 1 / 0 9 / 2 3 / h e a l t h - e q u i t ytourists-white-scholars-colonizing-health-disparitiesresearch/ Mertens, D. M. (2012) Transformative mixed methods: Addressing inequities. American Behavioral Scientist, 56(6), 802–813. https://doi.org/10.1177/ 0002764211433797 Mertens, D. M. (2013). Emerging advances in mixed methods: Addressing social justice. Journal of Mixed Methods Research, 7(3): 215–218. https:// doi.org/10.1177/1558689813493994 National Institutes of Health. (2021, October 29). Final NIH policy for data management and sharing. https://grants.nih.gov/grants/guide/notice-files/ NOT-OD-21-013.html Peller, G. (1990). Race-consciousness. Duke Law Journal, 1990(4), 758–847. https://doi.org/10.2307/ 1372723 Pett, M. (2015). Nonparametric statistics for health care research: Statistics for small samples and unusual distributions. Sage. Rivas-Drake, D., Camacho, T. C., & Guillaume, C. (2016). Just good developmental science: Trust, identity, and responsibility in ethnic minority recruitment and retention. Advances in Child Development and Behavior, 50, 161–188. https:// doi.org/10.1016/bs.acdb.2015.11.002 Shannon-Baker, P. (2021). Centering race in mixed and multimethod research on implicit bias: a systematic review. International Journal of Multiple

INTEGRATING SECONDARY DATA FROM ETHNICALLY AND RACIALLY MINORITIZED GROUPS 371

Research Approaches, 13(1). 55–73. https://doi. org/10.29034/ijmra.v13n1a3. Smith, E. (2008). Pitfalls and Promises: The Use of Secondary Data Analysis in Educational Research. British Journal of Educational Studies, 56(3), 323–339. http://www.jstor.org/stable/20479605 Solorzano, D. G., & Bernal, D. D. (2001). Examining transformational resistance through a critical race and latcrit theory framework: Chicana and Chicano students in an urban context. Urban Education, 36(3), 308–342. https://doi.org/10.1177/ 0042085901363002 Syed, M., Santos, C., Yoo, H.C., & Juang, L.P. (2018). Invisibility of racial/ethnic minorities in developmental science: Implications for research and institutional practices. American Psychologist, 73(6), 812–826, https://doi.org/10.1037/amp0000294 Taffe, M. A., & Gilpin, N. W. (2021). Equity, diversity and inclusion: Racial inequity in grant funding from the U.S. National Institutes of Health. Published in eLife. 18 January. https://elifesciences.org/articles/65697 Tao, T. (2021). “Doing justice” in psychological methodology: From science and experiments to anecdotes. New Ideas in Psychology, 61, 1–8. Wang, A. C., Mac, J., & Museus, S. (2021). The power of ethnic studies at Asian American and

Native American Pacific Islander serving institutions. About Campus: Enriching the Student Learning Experience, 26(1): 10–16. https://doi. org/10.1177/1086482220953164 Watkins, D.C. (2022). Secondary data in mixed methods research. Sage. Watkins, D.C., & Gioia, D. (2015). Mixed methods research. Oxford University Press. Watkins, D.C., Johnson, N.C., 2023. Advancing education research through mixed methods with existing data. In: Tierney, R.J., Rizvi, F., Erkican, K. (Eds.), International Encyclopedia of Education, vol. 12. Elsevier. https://dx.doi.org/10.1016/B978-0-12-818 630-5.11064-4. Williams, D. R., Yu, Y., Jackson, J. S., & Anderson, N. B. (1997). Racial differences in physical and mental health: Socio-economic status, stress, and discrimination. Journal of Health Psychology, 2(3), 335–351. https://doi.org/10.1177/135910539 700200305 White House, 2021. Fact Sheet: U.S. Efforts to Combat Systemic Racism. 21 March. Statements and Releases. www.whitehouse.gov/briefingroom/statements-releases/2021/03/21/factsheet-u-s-efforts-to-combat-systemic-racism/

26 Beyond the Joint Display in Mixed Methods Convergent Designs: A Case-Oriented Merged Analysis Carolina Bustamante

INTRODUCTION How to merge qualitative and quantitative data has been regarded as one of the most challenging elements to execute in mixed methods research (Guetterman et al., 2015). In convergent designs, merging data includes strategies for “merging the results, assessing whether the results from the two databases are congruent or divergent, and, if they are divergent, then analyzing the data further to reconcile the divergent findings” (Creswell & Plano Clark, 2011, p. 223). One of the most popular strategies to integrate data in a mixed methods convergent design is a joint display, a table or figure in which data from both the quantitative and qualitative strands are displayed together, allowing for the comparison of the two databases. Joint displays have gained the attention of mixed methods researchers in the last few years because they are an effective way to visualize data sources and findings, and understand integration (Bustamante, 2019; Fetters & Molina-Azorin, 2019; Guetterman et  al., 2015; Johnson et  al., 2019; Ling & Pang, 2021). This chapter focuses on another option for integration that goes beyond the comparison of the two databases in a joint display and may entail a more complex process: a case-oriented merged analysis (Creswell & Plano

Clark, 2011). For a discussion of how design typologies evolved over time, see also Chapter 2 (this volume). It consists of the selection of particular cases of interest or instantiations of the case (Schoonenboom, 2019) to further explore and analyze, extending the findings that a joint display might expose. Looking at outstanding cases engages researchers with difference, a concept described by Greene: “[a] mixed method way of thinking seeks not so much convergence as insight; the point is not a well-fitting model or curve but rather the generation of important understandings and discernments through the juxtaposition of different lenses, perspectives and stances” (2005, p. 208). However, there is little current guidance to conduct this type of analysis within a mixed methods convergent study to achieve the goal of describing and understanding cases. In order to address this gap in the literature, the purpose of this chapter is to exemplify a caseoriented merged analysis in a case study-convergent mixed methods study using the Technological Pedagogical Content Model (TPACK; Mishra & Koehler, 2006) as a theoretical lens. More specifically, a case study of professional development on Web 2.0 technologies for teachers of Spanish is used as an illustration. First, qualitative results from interviews, observations and documents and

BEYOND THE JOINT DISPLAY IN MIXED METHODS CONVERGENT DESIGNS

quantitative survey results were integrated via a TPACK-based joint display (Bustamante, 2019), facilitating the analysis for congruences and discrepancies in both sets of results for the whole group of participants. After this data-merging procedure, a case-oriented merged analysis was conducted. Quantitative demographic and survey data were used to select exceptional cases or the participants with the highest knowledge growth in order to further analyze qualitative data for those specific teachers. In combination with visual displays, this innovative approach to merging data brought novel insights into what makes participants’ learning and integration of technology the most effective. This chapter describes an illustrative case study-convergent mixed methods study, theoretical model and methods, followed by a detailed explanation of the case-oriented merged analysis and corresponding case-oriented visual displays, and analysis of integrated results. Last, the role of case-oriented merged analyses in data integration and new perspectives are discussed.

INTRODUCING A CASE-ORIENTED MERGED ANALYSIS ILLUSTRATIVE EXAMPLE: A CASE STUDY-CONVERGENT MIXED METHODS STUDY OF PROFESSIONAL DEVELOPMENT OF WEB 2.0 FOR TEACHERS OF SPANISH Background to the Illustrative Example Study Based on an eminent need for the integration of technology in world language classrooms in the United States (Gray et  al., 2010; Moore-Hayes, 2011), the University of Nebraska-Lincoln secured funding through a Teacher Quality Grant (Postsecondary Commission on Education & US Department of Education) to develop an online professional development program for teachers of Spanish for grades 7–12. The purpose of the program, called Web 2.0 for Teachers of Spanish, was to help teachers become effective practitioners on integrating Web 2.0 tools into the Spanish classroom. Participants were spread across different urban and rural areas of the state of Nebraska. The program was organized by 16 weekly modules, in which participants learned about and created products—in the same way that their students would—using a variety of web-based tools. The program had three basic goals for the teacherparticipants: to become technology literate with Web 2.0, to expand Spanish language and culture

373

knowledge, and to create lessons that reflect best pedagogical practices. Therefore, a case studyconvergent mixed methods study was used with the purpose of investigating a technology, pedagogy and content-based professional development on Web 2.0 technologies for teachers of Spanish and, consequently, integration of technology in the classroom. It sought to describe the experiences of the participants, as well as measure the outcomes of the program in order to enhance the description of the case. This research answered the question: To what extent do the qualitative results from the participants’ experiences confirm the measurement data on participants’ knowledge and technology integration? (Bustamante, 2019).

Theoretical Lens The Technological Pedagogical Content Knowledge (TPACK) model (Mishra & Koehler, 2006) is a framework for technology integration and teacher knowledge that integrates and emphasizes the interactions between and among technology, pedagogy and content. It argues that for effective integration of technology into the classroom, teachers’ experiences with technology need to be discipline specific. The components of this model not only served as the guiding constituents for the professional development program, but also as the theoretical lens that guided the data analysis in this study (Figure 26.1). Figure 26.1 consists of a three-fold Venn diagram that represents the three dimensions of TPACK—content, pedagogy and technology—in each of the three circles. The overlapping of the figures represents the intersection between areas. For example, when “Content Knowledge” (C) overlaps with “Pedagogical Knowledge” (P), the intersection of these two dimensions results in “Pedagogical Content Knowledge”. As represented in the middle point, the intersection among the three dimensions represents “Technological Pedagogical Content Knowledge”. The “Web 2.0 for Teachers of Spanish” program aimed at developing participants’ TPACK, by integrating “Spanish Language & Culture” (C), “Foreign Language Pedagogy” (P) and “Web 2.0” (T), as portrayed in Figure 26.1.

Qualitative, Quantitative and Mixed Methods Research Questions This research was guided by the following qualitative questions: (1) What are the experiences of

374

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 26.1  The Web 2.0 for Teachers of Spanish program and the TPACK model (Bustamante, 2014; adapted from Mishra & Koehler, 2006, p. 1025) the participants during the program? (2) What are the experiences of the participants integrating technology in their classrooms? In addition, the quantitative research questions were: (1) Are there significant differences in participants’ technology, pedagogy and content knowledge from pre- to post- to follow-up measures? (2) Are there significant differences in participants’ technology integration into their classrooms from pre- to post- to follow-up measures? These questions led to the aforementioned mixed methods research question: To what extent do the qualitative results from the participants’ experiences confirm the measurement data on participants’ knowledge and technology integration? (Bustamante, 2019).

Mixed Methods Research Design and Data Collection and Analysis Procedures Eighteen teacher-participants within a varied range of age, geographical location and years of teaching experience took part in this case study. A case study is “an empirical method that

investigates a contemporary phenomenon (the ‘case’) in depth and within its real-world context” (Yin, 2018, p. 15). A convergent mixed methods design was used (Creswell & Plano Clark, 2011), which involved collecting and analyzing qualitative and quantitative data concurrently for further integration. More specifically, this is a case study-convergent mixed methods study, in which “researchers employ a parent case study that includes a nested mixed methods design” (Guetterman & Fetters, 2018, p. 902). The procedural diagram (Figure 26.2) shows the qualitative and quantitative strands separately in rectangles at the top in white and black respectively. The procedures to the right are distributed based on point-in-time according to the timeline to the left of the diagram. Merging of strands is depicted using ovals at the bottom and in grey colour, to visually indicate mixing (black + white = grey). Qualitative strand. Multiple data sources were used for analysis. Data from interviews at the end of the professional development course, observations in the participants’ classrooms the semester following and documents with reflections during the experience were gathered. Data collection occurred at different points during the study: before, during

BEYOND THE JOINT DISPLAY IN MIXED METHODS CONVERGENT DESIGNS

375

Figure 26.2  Procedural diagram (Bustamante, 2014) and after the program. Thematic analysis was conducted to analyze the qualitative data. Quantitative strand. Two instruments were used to measure the participants’ knowledge in technology, pedagogy and content, and consequent integration of technology in their classrooms: The Adapted Chinese Language Teaching Institute survey (ACLTI) (Moeller et  al., 2011) and the TPACK survey (Schmidt et al., 2009). A repeated measures ANOVA was conducted to determine significant differences from the beginning of the program to the end to the following semester when

teachers were implementing the technologies learned in their Spanish classrooms. Qualitative and Quantitative Data Integration. The two data integration procedures used in this study are described in the sections that follow. First, I describe a merged data analysis comparison using a joint display, which led to the featured integration strategy for this chapter, a case-oriented merged analysis (Creswell & Plano Clark, 2011). A joint display was developed to determine concordance and discrepancies in technology, pedagogy and content between the two data sets

376

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

for the whole group of participants, using quantitative survey scales and corresponding quantitative themes and quotes. The components of the TPACK model, also part of the TPACK survey, were used both for grouping of the scales in the ACLTI instrument, as well as the grouping of codes and themes in the qualitative analysis. This process led to integration and to the development of a circular joint display with TPACK’s three-fold Venn diagram as the basis (Figure 26.3). Beginning in the middle and in black, the scales from the TPACK survey (Schmidt et  al., 2009) are represented in each component of the TPACK model. Four concentric circles grow around it: the first ring, also in black, includes the scales from the ACLTI survey and significance values, indicating growth from the baseline in

participants’ knowledge of technology, pedagogy and content. The second and third rings, in white, include themes and quotes from the qualitative data in each TPACK-based category. Scales that produced insignificant results from the baseline (pre-test measure) and negative qualitative findings are indicated in a line pattern to represent lack of congruency. The fourth and final ring, in grey, indicates the convergence of the qualitative and quantitative results, based on the fit of data integration (Fetters et al., 2013). Consistent with the procedural diagram, black represents quantitative data, white represents qualitative data and grey represents the mixing of black and white—in this case, integration of both strands of data (for a detailed description of the development of the joint display, please see Bustamante, 2019).

Figure 26.3  TPACK-based joint display (Bustamante, 2019)

BEYOND THE JOINT DISPLAY IN MIXED METHODS CONVERGENT DESIGNS

Based on the joint display analysis, the merging of results culminated in confirmation, expansion and discordance (Fetters et  al., 2013) at diverse points. Regarding “pedagogy”, the benefits of the professional development program were positively discussed by the participants, as reflected in all themes and corresponding quotes in the qualitative data. Confirmation of qualitative and quantitative results was evident in all survey scales, providing evidence for the effectiveness of the professional development program. The only exception was the scale of pedagogical beliefs, which reported no significant growth. A further look into the survey revealed issues related to the wording of these items in the scale. The learning reported in the qualitative data is based on the relationship between pedagogy and technology, but the items in the scale did not include any technology-related words. In this case, the integration of data was beneficial to illuminate problems in a quantitative instrument. Regarding “technology”, participants reported positive learning experiences in the qualitative data, as reflected in all but one theme in the joint display. All quantitative scales revealed growth, resulting in confirmation. However, the integration of results revealed expansion in the area of technology integration in the teachers’ classroom, because participants had both positive and negative experiences regarding technology access in the schools. Despite the effectiveness of the professional development program preparing participants to integrate technology in their classes, the large variation in technology access in the schools, evident in the qualitative results, affected technology integration in the classroom. These results illuminated an issue that needed further exploration or expansion. Regarding “content”, the merging of results revealed both expansion and discordance. First, in the area of Spanish proficiency, the qualitative results indicated that the program not only helped the participants improve their teaching practices, but also their Spanish language skills. Significant growth was also reported in the quantitative scores. However, inconsistent use of spoken Spanish by the teachers was observed in the classroom. The integration of data illuminated another area worthy of exploration in future studies in order to expand the understanding of improvement of language proficiency as a result of participation in a professional development program. Second, in the area of professional community, the qualitative results indicated the development of a professional community as a very important aspect of the program. However, the quantitative scale from the ACLTI survey reported no significant growth in this area, resulting in discordance. Upon further examination of the items in the scale, there was

377

no mention of the community specifically created with the colleagues from the professional development program. Participants could have interpreted the wording from these items as a community with teachers in their respective schools. Again, integration of data illuminated problems with the quantitative instrument.

BEYOND THE JOINT DISPLAY WITH CASE-ORIENTED MERGED ANALYSIS The Need for a Case-Oriented Merged Analysis After the merged data analysis using the joint display was conducted, an additional question emerged: What conditions make technology learning and integration the most effective? Whereas the joint display allowed for an effective comparison of the qualitative and quantitative databases for the whole group of participants, it did not bring insights into individual exceptional cases of learning and integration of technology. Therefore, a case-oriented merged analysis (Creswell & Plano Clark, 2011) was performed. It involved selecting the participants whose results in the scales from the ACLTI survey represented the largest growth in technology, pedagogy and content knowledge. Developing a case begins with detecting a phenomenon that will be the target of research (Schoonenboom, 2019). The purpose of this case-oriented merged analysis was to determine who was impacted the most by the Web 2.0 for Teachers of Spanish program, and further explore their specific experiences in the qualitative data because that might bring insights into what makes participants’ learning and integration of technology the most effective. A challenge in studying cases is the amount of data available and decision on what quantitative and qualitative data to analyze and how to do it (Schoonenboom, 2019). In a study conducted in rural Kenya by Glewwe et  al. (2009), it was found that using textbooks did not have an effect on test scores for the students in 50 schools. In an analysis of the mixing data purposes of this study, Schoonenboom et al. (2018) explained that one student per school with a median score was selected and interviewed, and asked to read a selection from the textbook. Researchers found that most median students were unable to read or answer the questions because the textbooks were written in English, and this was not the students’ first language. However, the study did not end at that point. The researchers decided to analyze the quantitative data further and

378

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

focus on the high achievers, and found that textbooks did have an effect on this group. In the case of the illustrative study for this chapter, a similar process occurred. A merged analysis for the whole group of participants on TPACK using a joint display was followed by a focus on “high achievers” or exceptional cases, consisting of the participants most impacted by the professional development program.

Selection of Exceptional Cases for the Illustrative Case-Oriented Merged Analysis Within the quantitative data, multiple pre-post measures were available for each participant. In

order to select the most improved teachers, a series of steps was used: (a) individual survey scores from the participants in the nine scales from the pre- and post-test were retrieved, together with the difference between the two measures; (b) for each scale, participants whose score was above the mean difference for the group were selected; and (c) teachers who were above such mean in four or more scales were determined to be the participants who received the greatest impact from the professional development program. Four was determined as the minimum number because there were multiple participants who were above the mean in two and three scales. Five teachers fell within the criteria for selection of exceptional cases, meaning that they were above the mean in four or more scales in the ACLTI survey (Tables 26.1, 26.2 and 26.3).

Table 26.1  Mean and individual differences for pedagogy-related scales (Bustamante, 2014). Subject

Laurie Brandy Kayla Kim Omar

P1 Classroom practices

P2 Assessment

P3 Beliefs

P4 Methodologies

MD

Individual difference

MD

Individual difference

MD

Individual difference

MD

Individual difference

4.647 4.647 4.647 4.647 4.647

5.00 9.00 16.00 17.00 1.00

1.647 1.647 1.647 1.647 1.647

3.00 6.00 2.00 7.00 5.00

1.53 1.53 1.53 1.53 1.53

5.00 −3.00 1.00 7.00 8.00

6.41 6.41 6.41 6.41 6.41

13.00 16.00 1.00 6.00 12.00

Source: Bustamante (2014).

Table 26.2  Mean and individual differences for technology-related scales (Bustamante, 2014) Subject

Laurie Brandy Kayla Kim Omar

T1 Beliefs

T2 Proficiency

T3 Integration

MD

Individual difference

MD

Individual difference

MD

Individual difference

3.235 3.235 3.235 3.235 3.235

5.00 8.00 3.00 7.00 7.00

61.41 61.41 61.41 61.41 61.41

89.00 56.00 73.00 85.00 46.00

33.882 33.882 33.882 33.882 33.882

29.00 29.00 57.00 46.00 17.00

Source: Bustamante (2014).

Table 26.3  Mean and individual differences for content-related scales (Bustamante, 2014) Subject

Laurie Brandy Kayla Kim Omar

C1 Spanish proficiency

C2 Professional community with Spanish teachers

MD

Individual difference

MD

Individual difference

1.471 1.471 1.471 1.471 1.471

4.00 1.00 4.00 3.00 0.00

–0.12 –0.12 –0.12 –0.12 –0.12

–4.00 1.00 4.00 5.00 –1.00

Source: Bustamante (2014).

BEYOND THE JOINT DISPLAY IN MIXED METHODS CONVERGENT DESIGNS

Tables 26.1, 26.2 and 26.3 are based on the ACLTI survey and each portray one of the three categories within TPACK: pedagogy (P1, P2, P3 and P4 scales), technology (T1, T2 and T3 scales) and content (C1 and C2 scales) accordingly. These nine scales were used to determine the mean difference for the whole group of participants (MD), as well as the “Individual Difference” for each participant in scores between the pre- and posttests. The “Subjects” included in these tables are “Laurie”, “Brandy”, “Kayla”, “Kim” and “Omar” because they were above the MD in four or more scales. Laurie’s individual differences were above the MD in P1, P2, P3, P4, T1, T2 and C1 for a total of seven scales. Brandy’s individual differences

379

were above the MD in P1, P2, P4, T1 and T2 for a total of five scales. Kayla’s individual differences were above the MD in P1, P2, T2, T3, C1 and C2 for a total of six scales. Kim’s individual differences were above the MD in P1, P2, P3, T1, T2, T3, C1 and C2 for a total of eight scales. Omar’s individual differences were above the MD in P2, P3, P4 and T1 for a total of four scales. Furthermore, differences and commonalities among the selected improved participants were determined based on demographic data: gender, age, geographical location and years of teaching experience. In this case, the demographic data were useful to make distinctions among those five most impacted participants (Figure 26.4).

Figure 26.4  Participants’ demographic data

In Figure 26.4, the relative sizes of the areas in the pie graphs correspond to the percentage of individuals for each category. Out of the eighteen participants in the study, 82 per cent were female and 18 per cent male. Regarding “Age”, 35 per cent were between 20 and 29 years old, 35 per cent were between 30 and 39 years old, 24 per cent were between 40 and 49 years old, and 6 per cent were more than 50 years old. Regarding “Geographical Location”, 47 per cent worked in rural areas and 53 per cent in urban areas. Lastly, regarding “Years

of Teaching Experience”, 47 per cent had between 1 and 5 years, 12 per cent had between 6 and 10 years, 6 per cent had between 11 and 15 years, 24 per cent had between 16 and 20 years, and 11 per cent had between 21 and 25 years. The most salient characteristics were geographical location and age. The five participants were divided into two clusters, determining two instantiations of the case or particular contexts in which the case developed (Schoonenboom, 2019). The first cluster included Laurie, Brandy and Kayla, who

380

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

interestingly were the three oldest teachers in the entire group of participants (54, 42 and 44 years old respectively) and were located in rural areas; and

the second cluster included Kim and Omar, 35 and 29 years old respectively, who were located in the same urban school district (Figure 26.5).

Figure 26.5  Demographic data examination in case-oriented merged analysis

Figure 26.5 portrays the five selected participants within their corresponding areas in the pie charts for “Geographical Location” and “Age”. This helps to visually identify the differences between the two clusters in a clear way in those two categories. Teachers in rural areas and older in age are represented in white, and teachers in urban areas and younger in age are represented in black. In this and the following visuals (Figures 26.5, 26.6 and 26.7), the colours black and white are not related to quantitative and qualitative data as in the procedural diagram (Figure 26.2) and joint display (Figure 26.3). They are used instead to visually identify the two clusters of participants.

Corresponding Case-Oriented Visual Displays for Case-Oriented Merged Analysis Results Qualitative data that pertained specifically to these five exceptional participants were compared to the group’s experience. This process led to a detailed understanding and allowed for a “dialogue” or a “meaningful two-way exchange of information and inferences between varied types of sources

gathered”—the quantitative and qualitative data (Bazeley, 2018, p. 8). Figure 26.6 displays the three specific areas unfolding from the prior joint display in which notable differences were found in the qualitative data between the two clusters of participants: (1) “Classroom practices and assessment” within “Pedagogy”; (2) “technology integration” within “Technology”; and (3) “sharing knowledge”, a theme beyond technology, pedagogy and content, represented further to the right. This figure shows that merging of data was not limited to the joint display and whole group of participants analysis. It went beyond it and further analyzed individual participants in those categories. Figure 26.7 portrays a case-oriented visual that aligns the three continuums that unfolded from the joint display in Figure 26.6 (curved lines with arrows at both ends). These continuums represent each of the three categories, “Pedagogy, “Technology” and “Beyond T, P & C” (Beyond technology, pedagogy and content), indicated to the left. They are distributed from left to right into negative (“–”), positive (“+”) and exceptional experiences (“++”) found in the qualitative data. For the purpose of this analysis, negative experiences are defined as situations in which participants experienced difficulties during the

BEYOND THE JOINT DISPLAY IN MIXED METHODS CONVERGENT DESIGNS

381

Figure 26.6  Case-oriented merged analysis visuals unfolding from joint display

professional development program or the following school year, both in learning and/or integration of technology; positive experiences are defined as situations in which participants’ learning experiences resulted in improved teaching practices and/or an effective integration of technology in the classroom; lastly, exceptional experiences are defined as situations in which participants’ learning experiences resulted in unusually good, beyond positive improved teaching practices and/or effective integration of technology in the classroom. Consistent with Figure 26.5, teachers in rural areas and older in age are represented in white, and teachers in urban areas and younger in age are represented in black. Quotes from the qualitative data that portray the experiences from

the participants in both clusters are included for each continuum in this visual display. Regarding “pedagogy”, the teachers in the urban cluster had similar positive experiences to other participants in the study. They realized what was lacking in their classes, which led them to question their “textbook-based curriculum” and expand their “assessment” practices. However, Kayla and Brandy—in the rural cluster—reported an exceptional change in their curriculum and assessment approach, respectively, which was distinctively different from the rest of the group. Kayla did not order a Spanish textbook the school year following the professional development program. She designed her own curricula, including all web-based projects learned during the program.

382

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 26.7  Case-oriented merged analysis visual display

Brandy implemented a new assessment system in which tests did not have the largest weight, but instead the evaluations were task-based. These tasks used the three modes of communication (interpretive, interpersonal and presentational) via technology projects and an ePortfolio for

self-assessment. Out of the group of participants, Kayla and Brandy reported the most outstanding changes in pedagogical practices. Regarding “technology”, “technology access in school” was the area where notable differences were found in both clusters. On one hand, Kim

BEYOND THE JOINT DISPLAY IN MIXED METHODS CONVERGENT DESIGNS

and Omar had especially poor access to websites in their schools and consequently very limited access to Web 2.0 tools. They mentioned this in their interviews, reflective papers and surveys. Kim sought support from the administration multiple times, with no positive results. Omar also commented that access to equipment was difficult during testing time because the computer labs were assigned to the teachers of content areas that required standardized testing. On the other hand, Kayla and Brandy were two of the teachers with the greatest support from the school regarding technology. None of them had problems with blocked Web 2.0 sites. Kayla’s principal and superintendent were very excited about everything she was learning and implementing in her Spanish class. Brandy’s school was one of the few who had a one-to-one student–laptop ratio, which greatly facilitated the integration of these tools. Lastly, one more exceptional experience was found in the theme “sharing knowledge” within the “beyond technology, pedagogy and content” category. Again, Kim and Omar reported similar experiences to the rest of the participants regarding the ways they shared the knowledge they gained during the program with their colleagues at their schools: via email, at department meetings or at curriculum planning groups. On the contrary, Laurie and Kayla went beyond their schools and districts to share such knowledge. The semester following the professional development program, they prepared a session together on the use of technology in the classroom for the state’s annual conference for world language teachers. For Kayla, this represented a five-hour drive each way, which demonstrated how motivated she was. The session was a success. Figure 26.7 shows dashed grey lines that connect the experiences of the teachers in each cluster—urban and rural—across the aforementioned categories. Teachers in the rural cluster displayed exceptional experiences in the three categories, represented with a straight vertical line. This indicates that the large impact of the professional development program was reflected in their practices. However, the teachers in the urban cluster did not display exceptional or outstanding experiences in any area, compared to the rest of the participants. Moreover, their experience with technology access at the school was negative. This might indicate that the support that the teachers received from the administration regarding technology-based practices in the classroom had an influence on the teachers’ decisions about the curriculum. Compared to the teachers in the rural cluster, the lack of support by the administration “pulled” the urban teachers back in the continuum of experiences (represented with a bent

383

line towards the left). This is an area worthy of exploration in future studies. This analysis also questions the popular belief that younger teachers might be better, more interested or capable in learning and integrating technology.

Novel Insights Generated by a Case-Oriented Merged Analysis In this study, quantitative data were used to determine the overall impact of the Web 2.0 professional development program on a group of teachers of Spanish. These scores were contrasted with qualitative data via a joint display to determine the overall experience of the group. This merged analysis confirmed the effectiveness of the program in most areas. Discordance between the qualitative and the quantitative data illuminated issues with the survey scale items, and two areas of expansion were identified. However, the decision of conducting a case-oriented merged analysis allowed for inferences that went beyond the merged analysis conducted using the joint display. Further analyzing the same quantitative data to determine which participants had the most impact by the program and looking at demographic data revealed two instantiations of cases: integration of technology in rural and urban areas. Exploring in more depth the qualitative data for those specific participants revealed that the support that teachers receive from the district and school administration has an important impact in their curricular decisions. Interestingly, participants in rural areas received more support and resources than participants in urban areas and had more exceptional experiences. These conclusions were facilitated not only by selecting the participants with the most growth using the quantitative data and the further exploration of their particular experiences in the qualitative data, but also by using a caseoriented visual display to portray these experiences. Visualizing research involves strategies that researchers use to “understand, present, and frame research” (Wheeldon & Åhlberg, 2012, p.1). Therefore, visual displays in diagram, map or graph format aid to clarify intricate processes (Dickinson, 2010). They assist with displaying results or findings (e.g., Erwin et  al., 2011) and facilitate “communication, thinking, and learning” (Schnotz, 2002, p.101). In other words, graphic displays make complexity accessible by “combining words, numbers, and pictures” (Tufte, 2001, p. 180). For example, the conclusion that the support that teachers receive from the district and school administration may have

384

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

significant impact in their curricular decisions was made evident by visualizing the three continuums of experiences together and the connecting lines for each cluster. Representing the integration of qualitative and quantitative data might be difficult due to the lack of software applications that facilitate it (Guetterman et  al., 2015). Qualitative software such as MAXQDA® (www.maxqda.com/) includes mixed methods tools to create tables and figures to visually represent data integration. However, when a theoretical framework is also integrated, as in the case of the visual displays showcased in this chapter, a more graphics-oriented software is needed. PowerPoint® (http:// office.microsoft.com) has been identified as a useful tool to represent visual elements in mixed methods research—for example, procedural diagrams (Miller & Bustamante, 2016). In all figures included in this study, PowerPoint® was utilized to develop the procedural diagram, joint display and case-oriented merged analysis display using basic shapes, lines and fill effects.

IMPLICATIONS FOR CASE-ORIENTED MERGED ANALYSIS A case-oriented merged analysis is an effective method to further investigate particular or exceptional cases within a mixed methods convergent study, as in the illustrative example described above. It does not necessarily have to be conducted as part of a convergent design, it may work for other designs as well (for an explanation of case development in sequential mixed methods designs, read Schoonenboom, 2019). The study presented in this chapter offered an approach to identify exceptional or highest performance individuals using quantitative measures for further in-depth qualitative analysis: retrieving individual survey scores from participants from pre- and post-tests, together with the difference between the two measures; selecting participants whose score was above the mean difference for the group in four or more scales; and determining differences and commonalities among the exceptional cases based on demographic data. Depending on the phenomenon of interest for the case, lowest performance individuals can also be identified using a similar approach. Development of cases “also enable[s] and make[s] use of rich databases” (Schoonenboom, 2019, p.10). These procedures facilitated thickness and richness of data, or significance enhancement, a

rationale for mixing qualitative and quantitative data (Collins et  al., 2006). Selecting individual or exceptional cases using quantitative data and further analyzing the corresponding qualitative data brought insights that went beyond the results from the initial joint display, contributing to a more complete understanding of the case. Whereas the portrayal of the participants’ group experience allowed the researcher to determine the effectiveness of the professional development program, the selection and analysis of exceptional cases exposed the best conditions for learning and integration of technology. The latter would not have been possible without a case-oriented merged analysis procedure. In addition, visual displays result in better understanding of research. For a discussion of using software for innovative integration in mixed methods research, see Chapter 22 (this volume). There are few examples of case-oriented merged analysis and consequently case-oriented visual displays for mixed methods studies in the literature. This case-oriented merged analysis visual display helped to clearly identify the differences between the two clusters of participants, contributing again to a better understanding of the case. Also, the case-oriented merged analysis visual display in this study illustrated a more graphical way of organizing data beyond the typical table or matrix. Aligning the three continuums in a graphic and connecting the experiences of the participants with dotted lines allowed for further inferences of interest, such as determining the impact that the lack of support from the school administration had on teachers’ integration of technology. In sum, this case-oriented merged analysis illustrated a process that integrates quantitative outcome results with qualitative findings from multiple sources using visual displays and narrative. Specifically, it offered an approach for looking across multiple quantitative measures to identify exceptional/highest performing individuals for further in-depth qualitative analysis. In addition, the case-oriented merged analysis is a method to integrate key qualitative themes to better understand highest performers for a case comparison of different clusters of participants. The unique format of the visual display provided clarity on this case comparison and allowed for further conclusions on the phenomenon studied. It is the hope of the author that this work will inspire other researchers who use convergent mixed methods design to explore possibilities to integrate data beyond a joint display employing a case-oriented merged analysis and be open and creative in the visual representation of such integration.

BEYOND THE JOINT DISPLAY IN MIXED METHODS CONVERGENT DESIGNS

WHAT TO READ NEXT Guetterman, T. C., & Fetters, M. (2018). Two methodological approaches to the integration of mixed methods and case study designs: A systematic review. American Behavioral Scientist, 62(7), 900–918. http://dx.doi.org/10.1177/0002764218772641

This article provides clarification on mixed methods-case study design (mixed methods studies that include a nested case study for the qualitative strand) vs. case study-mixed methods design (case studies that include a nested mixed methods design), describing key methodological features through four exemplar interdisciplinary studies. This information was useful to define the correct terminology for the research design of the illustrative study in this chapter and it might be of interest to researchers considering case study as a research approach. Schoonenboom, J. (2019). Develop your case! How controversial cases, subcases, and moderated cases can guide you through mixed methods data analysis. Frontiers in Psychology, 10(1369): 1–13. http://dx.doi.org/10.3389/fpsyg.2019.01369

This article describes case development within mixed methods studies in detail through a variety of examples, both in convergent and sequential designs. It explains how cases can be developed across several studies and in mixed methods research, offering an approach that can be applicable to a range of studies. Schoonenboom, J., Johnson, R. B., & Froehlich, D. E. (2018). Combining multiple purposes of mixing within a mixed methods research design. International Journal of Multiple Research Approaches, 10(1), 271–282. https://doi.org/10.29034/ijmra. v10n1a17

This article describes how multiple purposes for data mixing can be identified and incorporated into research on a within-study basis rather than applying it to the entire study. “Some purposes are identified at the beginning of the study, and other purposes emerge during the conduct of the study” (p. 271), as was the case with the case-oriented merged analysis presented in this chapter.

REFERENCES Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage. http://dx.doi.org/ 10.4135/9781526417190

385

Bazeley, P., & Kemp, L. (2012). Mosaics, triangles, and DNA: Metaphors for integrated analysis in mixed methods research. Journal of Mixed Methods Research, 6(1), 55–72. http://dx.doi.org/ 10.1177/1558689811419514 Bustamante, C. (2014). Professional development of Web 2.0 for teachers of Spanish: A mixed methods case study. Doctoral dissertation. The University of Nebraska-Lincoln. Bustamante, C. (2019). TPACK and teachers of Spanish: Development of a theory-based joint display in a mixed methods research case study. Journal of Mixed Methods Research, 13(2), 163–178. https:// doi.org/10.1177/1558689817712119 Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L. (2006). A model incorporating the rationale and purpose for conducting mixed methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4, 67–100. Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Sage. Dickinson, W. (2010). Visual displays for mixed methods findings. In Sage handbook of mixed methods in social & behavioral research (pp. 469–504). Sage. www.doi.org/10.4135/9781506335193 Erwin, E. J., Brotherson, M. J., & Summers, J. A. (2011). Understanding qualitative metasynthesis: Issues and opportunities in early childhood intervention research. Journal of Early Intervention, 33(3), 186– 200. https://doi.org/10.1177/1053815111425493 Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designsprinciples and practices. Health Services Research, 48(6, Pt 2), 2134–2156. http://dx.doi.org/10.1111/ 1475-6773.12117 Fetters, M. D., & Molina-Azorin, J. F. (2019). In this issue: Innovations in mixed methods—causality, case study research with a circular joint display, social media, grounded theory, and phenomenology. Journal of Mixed Methods Research, 13(2), 123–126. https://doi.org/10.1177/1558689819834986 Glewwe, P., Kremer, M., and Moulin, S. (2009). Many children left behind? Textbooks and test scores in Kenya. American Economic Journal: Applied Economics, 1, 112–135. http://dx.doi.org/10.1257/ app.1.1.112 Gray, L., Thomas, N., & Lewis, L. (2010). Teachers’ use of educational technology in U.S. public schools: 2009 (NCES 2010-040). Retrieved from: http://nces.ed.gov/pubs2010/2010040.pdf Guetterman, T., Creswell, J. W., & Kuckartz, U. (2015). Using joint displays and MAXQDA software to represent the results of mixed methods research. In Use of visual displays in research and testing: Coding, interpreting, and reporting data. 145–175. Information Age Publishing.

386

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Guetterman, T. C., & Fetters, M. (2018). Two methodological approaches to the integration of mixed methods and case study designs: A systematic review. American Behavioral Scientist, 62(7), 900–918. http://dx.doi.org/10.1177/0002764218772641 Johnson, R. E., Grove, A. L., & Clarke, A. (2019). Pillar integration process: A joint display technique to integrate data in mixed methods research. Journal of Mixed Methods Research, 13(3), 301–320. https://doi.org/10.1177/1558689817743108 Ling, H. L., & Pang, M. F. (2021). A vignette-based transformative multiphase mixed methods interventional study featuring Venn diagram joint displays: Financial education with Hong Kong early adolescent ethnic minority students. Journal of Mixed Methods Research. https://doi.org/10.1177/ 1558689821989834 Miller, D. & Bustamante, C. (2016). Drawing mixed methods procedural diagrams. In A. J. Moeller, J. W. Creswell & N. Saville (Eds.), Second language assessment and mixed methods research (Studies in language testing, Volume 43). 84–118. Cambridge University Press. Mishra, P. & Koehler, M. J. (2006) Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. http://dx.doi.org/10.1111/j.1467-9620. 2006.00684.x Moeller, A., Plano Clark, V. L., & Hustad, A. (2011). Chinese Language Teaching Institute survey. Unpublished instrument, University of NebraskaLincoln. Moore-Hayes, C. (2011). Technology integration preparedness and its influence on teacher-efficacy.

Canadian Journal of Learning and Technology. 37(3). http://dx.doi.org/10.21432/T2B597 Schmidt, D. A., Baran, E., Thompson A. D., Koehler, M. J., Mishra, P. & Shin, T. (2009). Technological pedagogical content knowledge (TPACK): The development and validation of an assessment instrument for preservice teachers. Journal of Research on Technology in Education, 42(2), 123–149. http://dx.doi.org/10.1080/15391523. 2009.10782544 Schnotz, W. (2002). Commentary: Towards an integrated view of learning from text and visual displays. Educational Psychology Review, 14, 101–120. https://doi.org/10.1023/A:1013136727916 Schoonenboom, J. (2019). Develop your case! How controversial cases, subcases, and moderated cases can guide you through mixed methods data analysis. Frontiers in Psychology, 10(1369): 1–13. http://dx.doi.org/10.3389/fpsyg.2019.01369 Schoonenboom, J., Johnson, R. B., & Froehlich, D. E. (2018). Combining multiple purposes of mixing within a mixed methods research design. International Journal of Multiple Research Approaches, 10(1), 271–282. https://doi.org/10.29034/ijmra. v10n1a17 Tufte, E. (2001). The visual display of quantitative information (2nd ed.). Graphics Press. Wheeldon, J. & Åhlberg, M. K. (2012). Mapping mixed-methods research: theories, models, and measures. In Visualizing social science research: Maps, methods, & meaning (pp. 113–148). Sage. www.doi.org/10.4135/9781483384528 Yin, R. (2018). Case study research and applications: Design and methods (6th ed.). Sage.

The Untapped Potential of Technology for Integration: Section 4 Conclusions Timothy C. Guetterman

The chapters in Section 4 present unique and innovative approaches to integration in mixed methods research. By providing practical illustrations and advice on achieving integration with thoughtful technology applications, the chapter authors of Section 4 move the field forward. With novel ways of embedding technology into integration, the chapters expand our collection of integration strategies and build our contemporary integration methods. Software can be useful in many ways, from helping to facilitate the integration of qualitative and quantitative perspectives to using visualizations and existing datasets. Gamebased research has opened up new ways of using technology to mediate data collection participation. While chapters in other Handbook sections might refer to technology applications, the unique contribution of the curated collection of chapters that comprise Section 4 is the diverse ways in which authors have applied technology to enhance their integration efforts across a variety of settings. Although this collection cannot be considered representative of all the possibilities, it serves as a launching place for others to use these ideas to accelerate the use of software for integration. In this Conclusion to Section 4, I outline future directions to promote the use of technology in mixed methods research. My hope is that future mixed methods research design will infuse

technology in ways that are appropriate for the study and its context. I see advocacy roles for mixed methods research practitioners in the field, those who act as methodologists tasked with advancing practices, and software developers who are working on new ways to embed technology in our mixed methods research design practices.

FUTURE DIRECTIONS Mixed Methods Researchers Technology can be helpful in addressing mixed methods research questions or aims by providing ways to conceptualize integration, tools to conduct integrative analyses and means to represent integration. For mixed methods researchers, it may involve a careful review of literature that describes the use of technology. Among the challenges is simply that discussions of the use of technology in empirical articles is woefully lacking at this time. I imagine a future where researchers are more explicit about their intended role for technology, and their design and reporting decisions. In brief, as others see descriptions of technology use more explicitly, it will likely inspire this practice.

388

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

When designing a mixed methods study, consider what technology may be helpful. First, it will likely help to identify aims, research questions, data sources and integration strategies. Then, I suggest exploring software options by reviewing what is available and using trial versions to practice and decide which software will be most helpful in achieving aims and addressing research questions. Furthermore, in writing proposals, discussing the use of software or specialized technology when describing the research approach will help both the researcher to conceptualize a plan for integration and help the reader or reviewer understand the plan. Researchers might mention a plan to use a particular software for mixed methods analysis, just as they would mention a plan to use a particular statistical software program or qualitative program. Mixed methods software includes traditionally qualitative programs that have added mixed methods features, traditionally statistical programs that have added text analysis features, and specialized software for natural language processing and text mining (Chapter 23, Inaba & Kakai). If the software needs to be acquired, I also expect to see the software included in the budget. For students preparing proposals, consider what software might be already available through your institution or available at a substantial student discount. A deciding factor for me to use software has been how software can support visualizations. I suggest considering what visuals will be useful for the process of integration and to provide evidence of integration. For example, visualization software tools may be helpful for developing procedural diagrams of the entire mixed methods design. Researchers have used visual software such as Microsoft Visio, draw.io, or even Microsoft PowerPoint or Google Slides to draw procedural diagrams that map out the flow of procedures and points of integration. The diagram can further include integration methods and software within the procedures enumerated. Thus, these visualization tools and drawing programs are particularly helpful in designing for integration. In reporting mixed methods research through articles, theses, dissertations and other reports, I also suggest discussing software that has been used for mixed methods integration. Again, software would likely appear in the methods section, such as when describing analysis. Perhaps more salient, is the use of the software to facilitate integration (Chapter 22, Kuckartz & Rädiker). An increasing number of software programs incorporate both qualitative and quantitative data and allow linking data to participants. These features facilitate additional analyses, such as cross-tabulating themes-by-statistics or comparing qualitative and quantitative results. In addition to these

programs, technology can also be useful for developing joint displays that include unique visuals, such as graphs, figures, charts, images or diagrams (Guetterman et  al., 2021). For example, Chapter 24, Brevik and Chapter 26, Bustamente include innovative visualizations of integration developed through specialized software. As I have argued elsewhere, the inclusion of additional visuals in joint displays is perhaps the next frontier to innovate integration through the joint display method (Guetterman et al., 2021). A potential future direction is reporting mixed methods results with advanced visualizations that go beyond the two-dimensional representation of joint displays in traditional printed (or PDF equivalent) journal articles, dissertations and theses. At a simple level, visualizations produced through graphic design software could lead to easy-to-digest infographics of mixed methods results. Thinking about a more complicated direction, consider that many mixed methods research studies are complex, but reporting is limited to a single layer of presentation. What if a journal article had an accompanying digital presentation of results that allowed readers to click through multiple layers? Although articles, theses and dissertations are increasingly including multiple joint displays, a digital, dynamic joint display would allow additional options. Through a dynamic joint display, a reader might be able to click through multiple hierarchical levels of analysis (e.g., compare qualitative and quantitative results and metainferences about physicians, and then move to the level of their patients). A dynamic joint display could allow visualizing different groups or categories, such as examining the results of one school district relative to another or focusing on one-at-atime. Finally, dynamic joint displays might be able to include more visuals such as videos, images or animations that are simply not possible in print.

Methodologists Methodologists are in a unique position to advance guidance about the implications of technology for integration and accelerate mixed methods researchers in achieving meaningful integration (Bazeley, 2018). Methodologists could develop the body of literature about technology in integration in at least three ways: (1) providing more illustrations of the use of software in mixed methods integration; (2) describing the intellection process of integration with visual joint displays; and (3) discussing the use of software in mixed methods teams. Chapter 22, Kuckartz & Rädiker, and Chapter 23, Inaba & Kakai, provided clear

SECTION 4 CONCLUSIONS

illustrations of technological features. Additional articles may help mixed methods researchers to conceptualize a path forward using software for integration by giving concrete steps, screenshots and advice on the use of software. Based on my personal experience providing software training, I find many mixed methods researchers are unaware of mixed methods features, even in programs they use regularly. A final issue is awareness of the features, but a need for helpful guidance in how to actually use the features. Additional methodological articles on software and technology may give investigators a starting point. Another major innovation in this section was the use of visualizations to help with the process of integration (see Chapter 22, Kuckartz & Rädiker; Chapter 23, Inaba & Kakai; and Chapter 24, Brevik). Visualizations are made possible through technology. However, we need more step-by-step articles to provide researchers with guidance to include innovative visuals in joint displays. For example, Haynes-Brown and Fetters (2021) illustrate the analytical process to achieve integration using bar graphs in joint displays. Fetters and Guetterman (2021) discussed the development of a joint display that included boxplots to facilitate integration and identify meta-inferences. A potential future direction is the use of technology in research teams. With large-scale studies, such as those using secondary data (e.g., Chapter 25, Watkins & Johnson), mixed methods research often occurs in interdisciplinary teams. While a small body of literature has addressed mixed methods teams (Curry et  al., 2012, Guetterman et  al., 2020), little has discussed software within the context of teams. Software with mixed methods features often includes team functions, yet the processes to use software as a team are often ad hoc. Any of these areas and more are ripe for development in methodological literature.

Software Developers Software developers are key influencers and want their software to be as useful and accessible as possible for researchers. As noted, traditionally qualitative programs have been increasing their mixed methods features, such as importing variables, conducting statistical analysis, transforming qualitative data into quantitative, and examining patterns among quantitative and qualitative results. Over the past decade, I have seen numerous advances, such as the ability to import more varied file formats and types of data (e.g., pictures, videos, structured survey data, social media data). Tools to visualize data and develop joint displays have also evolved.

389

Future developments might continue to develop these features into a robust set of analysis tools for integration. Integration tools that I would like to see include: features to develop mixed methods procedural design diagrams; features to create joint displays based on data stored in the software; features to document meta-inferences within the software; and features for integration at multiple levels. Integration at multiple levels requires extra explanation. The majority of mixed methods software features allow integration at the level of data (e.g., attaching variables to a transcript) or reporting (e.g., creating joint displays). However, sequential integration and integration of methods is largely absent. For example, software could be quite helpful in building integration where qualitative codes and themes are used to develop surveys and write items. Software could also facilitate connecting integration such as using quantitative results systematically to identify a purposeful sample. Currently, conducting these types of integrations systematically likely requires spreadsheets and word-processing software. My vision is to use software for all types of integration and integrative analyses. Software developers might be reluctant to develop such tools and features that may not be widely used. This likely compounds with many mixed methods researchers’ lack of awareness of how software can benefit. This impasse may be overcome through increased features in software, methodological articles about how to use the features and robust mixed methods training that includes the use of software.

CONCLUSION Consider what might be possible if software developers, methodologists and mixed methods research practitioners worked together to create something that any of them alone could not do. In other words, what if the field applied mixed methods thinking to the issue of technology by having individuals with different expertise collaborate, handle tensions, integrate and synergize views, and work towards a shared goal? Although this may seem like an audacious vision, realistic steps towards this goal are possible through grant-funded workgroups, conference sessions or conferences focused on technology and mixed methods. Technology can create new opportunities for integrating data types that we have not previously been able to manage or even envision.

390

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

REFERENCES Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage. Curry, L. A., O’Cathain, A., Plano Clark, V. L., Aroni, R., Fetters, M., & Berg, D. (2012). The role of group dynamics in mixed methods health sciences research teams. Journal of Mixed Methods Research, 6(1), 5– 20. https://doi.org/10.1177/1558689811416941 Fetters, M. D., & Guetterman, T. C. (2021). Development of a joint display as a mixed analysis. In The Routledge Reviewer’s Guide to Mixed Methods Analysis (pp. 259–275). Routledge. Guetterman, T. C., Abir, M., Nallamothu, B. K., Fouche, S., Nham, W. l. Nelson, C., Mendel, P. Forbush, B., Setodji, C. M., Kronick, S., Neumar, R. W., & Fetters, M. D. (2020). The process of team building among

content experts and methodologists: An example from an emergency medical services research investigation kick-off meeting. International Journal of Qualitative Methods, 19, 1–11. https://doi.org/ 10.1177/1609406920955117 Guetterman, T. C., Fàbregues, S., & Sakakibara, R. (2021). Visuals in joint displays to represent integration in mixed methods research: A methodological review. Methods in Psychology, 5, 100080. https://doi.org/10.1016/j.metip.2021.100080 Haynes-Brown, T. K., & Fetters, M. D. (2021). Using joint display as an analytic process: An illustration using bar graphs joint displays from a mixed methods study of how beliefs shape secondary school teachers’ use of technology. International Journal of Qualitative Methods, 20, 1609406921993286. https://doi.org/10.1177/1609406921993286

SECTION 5

Navigating Research Cultures in Mixed Methods Design

This page intentionally left blank

From Margin to Center: The Design Implications of a Cultural Component in Mixed Methods Research: Section 5 Introduction Elizabeth G. Creamer Section 5 of this handbook sets a precedent by dedicating a section to making explicit the ways specific mixed methods research practices are informed by cultural contexts and their influences. Our commitment to global inclusivity led us to invite a group of emerging, as well as more senior scholars geographically distributed around the globe, to contribute a chapter to this section. In addition to reflecting the global interest in mixed methods, the diverse perspectives advanced by the contributors offer a bridge across what Maxwell (2018) refers to as silos, and others have referred to as domains (e.g., Creamer, 2018; Creswell, 2010; Plano Clark & Ivankova, 2016) within the methodological literature about mixed methods. With Drs. Elizabeth G. Creamer and Elsa Lucia Escalante Barrios serving as section co-leads, contributors to Section 5 were invited to reflect on their experiences relative to two themes and to bring this cultural reflective lens to their illustrative example descriptions: (1) the application of sociocultural perspective in global research, and (2) the influence of wider professional contexts including (but not limited to) funding agencies and quality standards. A sociocultural or ecological framework involves the multiple layers of personal and interpersonal contexts that shape research practices (Plano Clark & Ivankova, 2016). An investigation with a sociocultural drive is most compatible with a

definition of mixed method research that emphasizes that its principal logic rests on the gains to understanding of engaging diverse perspectives (e.g., Creamer, 2018; Johnson et al., 2007). Culture is an elusive concept that is more readily grasped by those outside it than those deeply embedded in it (Gone & Kirmayer, 2010). Culture is generally defined in terms of shared language, values, beliefs and behavioral norms (Hitchcock et al., 2005). The cultural relevance of constructs is a particularly thorny issue in validating instruments developed in one culture and applied to another (Hitchcock et al., 2005). In an anthropological sense, culture is “shared, patterned, and historically reproduced symbolic practices that both facilitate and constrain meaningful human existence” (Gone, 2011, p. 235). In the title of a shape-shifting 1984 book, Black feminist author, bell hooks, used the expression “from margin to center” to describe how Black lives are hidden from mainstream American society and scholarship. “From margin to center” is an apt metaphor to compare how contributors in this section embedded culture in their research. Culture is generally at the margins of research or what Gone and Kirmayer (2010, p. 90) refer to “as a taken-for-granted background”. The purpose of this introduction is to orient the reader to the five chapters of Section 5 (see also

394

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table S5.1 1). It presents a figure to itemize the nine ways that a cultural component was acknowledged by chapter authors to influence mixed methods research design. The metaphor “from margin to center” provides a way to compare the impact of the cultural component on mixed methods research design across the chapters.

FROM THE MARGINS TO THE CENTER: THE IMPACTS NOTED OF CULTURE ON RESEARCH DESIGN Contributors to Section 5 heighten awareness of the many facets of culture that can be explored in research. For example, in Chapter 28, Hatta and Chapter 31 Anguera et  al. approach culture in terms of the verbal and non-verbal behavior involved in interactions with a professional clinician. In Chapter 30, Chu et al. describe an intervention designed to recognize differences in food practices among provinces in China. Within the context of evaluation in a US setting, Chapter 27, Hall and Boyce consider culture in terms of the impact of values embedded in an academic discipline. In Chapter 29, Chandanabhumma et al. take yet another approach to culture, equating it with the communities that were targeted in an intervention. For additional discussions of cultural influences in expanded mixed methods designs; see also the Introduction to Section 3 (this volume). The relevance of cultural to mixed methods research emerging from the health sciences is evident in that four of the five chapters in the section involve initiatives designed to address mental, physical and social health. These include two chapters, one by Anguera et al. (Chapter 31) and

a second by Hatta (Chapter 28), that both leverage mixed methods with observational research to better plot trends in the interactions between a health practitioner and a client. A third chapter by Chandanabhumma et al. (Chapter 29) was undertaken to address health disparities in access among community partners in the US. The fourth chapter, emerging from the context of the health sciences, by Chu et al. (Chapter 30) involves an initiative to reduce salt intake across diverse regions in China. Poth’s (2020) characterization of complex mixed method designs matches those encountered in Section 5, including that they occurred on some occasions in multiple, sometimes in geographically dispersed settings, involved diverse participants, were adaptive to changing conditions in the environment, and the predictability of the outcomes is low.

Conceptualizing How Culture Informs Research Design in Mixed Method Research Figure S5.1 uses the image of a pyramid to conceptualize the many faceted ways that chapter authors identified that culture informed their enquiry practices. When considered together, chapter authors associated a cultural lens with one or more of nine elements of the design and implementation of a mixed method study. These are: (1) in framing the purpose; (2) recognizing elements of the setting or location that influenced implementation; (3) pointing to the influence of external funding agencies and professional association guidelines or standards; (4) adopting transformative values or a social justice orientation; (5) incorporating a

Table S5.1  Summary of Section 5 Chapters: Navigating Research Cultures in Mixed Methods Design Chapter authors (country affiliation)

Chapter title

Jori N. Hall and Ayesha S. Boyce (USA) Taichi Hatta (Japan)

Culturally Responsive Mixed Methods Evaluation Design Integrating a Four-Step Japanese Cultural Narrative Framework, Ki-Shou-Ten-Ketsu, into a Mixed Methods Study Leveraging Mixed Methods Community-based Participatory Research (MMCBPR) in Diverse Social and Cultural Contexts to Advance Health Equity Cultural Diversity in Intervention Designs: A Chinese Illustrative Example Examining the Influences of Spanish Research Culture in Systematic Observation with Mixed Methods

P. Paul Chandanabhumma, Annika Agni and Melissa DeJonckheere (USA) Hongling Chu (China), Xuejun Yin (Australia), Huieming Liu (Australia) M. Teresa Anguera, Eulàlia Arias-Pujol, Francisco Molinero, Luca Del Giacco (Spain)

395

SECTION 5 INTRODUCTION

culturally sensitive theoretical or analytical framework; (6) pairing mixed methods with other research approaches sensitive to cultural nuances; (7) impacting the design of an intervention; (8) embedding elements of culture in the selection or construction of the instruments used for data collection; and (9) highlighting the need for reflexivity about cultural biases. Figure S5.1 is not a comprehensive catalog of all the ways a cultural component can influence research design in a mixed method study. The figure identifies facets of the research process recognized by the contributors in one or more of the chapters in Section 5. Integration is a noteworthy omission from Figure S5.1.

Orienting the Reader to Chapters in Section 5 of the Handbook The chapters in Section 5 illustrate that there are diverse ways that a cultural component can influence research design in a mixed methods study. As was the point of bell hook’s use of the phrase “from margin to center, many dimensions of culture are left unexplored in the chapters.

Culture is at the periphery of the way that Anguera et  al. describe a long-term research project involving mixed methods and systematic observation. These authors report on a series of studies that employed a structured observation protocol to capture interactions between a therapist and a young client. Framed as occurring within a Spanish context, Anguera et  al. employ the term “mixed methods” in the context of a multi-year research project that had qualitative and quantitative phases. Culture is central to the mixed method research reported by Chu et al. of a study reporting on the effects of an intervention funded by the World Health Organization that was designed to improve health by reducing salt intake across different provinces in China. Chu at al. single out several different ways that the research design was adapted to the cultural context. These include: (1) in establishing the purpose of the research to match the funding priorities of the external funder; (2) using input from regional experts to design the intervention; and (3) as a criterion that influenced the selection of team members. The influence of the cultural component is pervasive in the chapter submitted by Hatta. His reporting recognizes: (1) the Japanese setting as

Purpose

Theoretical Framework Social Justice Orientation

Setting

Research Methods

Reflexivity

Analytical Frameworks

Instrument Development

External Agencies

Figure S5.1  Summary of the ways a cultural component impacts research design among chapter authors Source: Author created.

396

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

context; (2) that he paired mixed methods with observation to understand the trajectory of the interactions between a patient and an oncologist in a medical setting; and that (3) he applied a “rhetorical-analytical framework” unique to Japan to analyze the data. The two US-based teams of authors contributing chapters take another tack to exploring the role of culture in their research, each with a transformative agenda. Chapter 27, Hall and Boyce incorporate culture in their reporting by (1) recognizing the wider sociocultural influence of professional guidelines in their definition of culture; (2) the purpose they establish to explore the intersections between identity and principles of the scientific culture in science and engineering disciplines; and (3) using values-engaged and equity-focused theories to inform the analysis. Chapter 29, Chandanabhumma et al. also tackle notions of culture in a unique way. These authors link it to a community-based participatory research project involving issues related to health equity and access. They define community in a way that overlaps culture in that it refers to individuals who share interests, identities, norms or needs. Their interest in culture impacted the research design in that engagement with community partners was woven throughout the project, from the conceptualization stage to data collection, data analysis and the dissemination of findings.

TENSIONS AND CHALLENGES IN METHODOLOGICAL WRITING ABOUT CULTURE We challenge members of the mixed method community to think of Section 5 as a global enterprise with its own discourse. The charge begins by decentering a US-based perspective by deliberately including investigators from across the globe. The wider challenge is to embed recognition of the cultural context in the design and implementation of a mixed method study in a meaningful way, including by adapting a culturally sensitive theoretical framework that can be applied to the instruments used for data collection and how the data are analyzed. Writing a methodological article that incorporates recognition of the sociocultural context is a challenging undertaking. It requires a type of code-switching for researchers long in the habit of writing empirical, rather than methodological, articles. It is no more readily accomplished by

those operating from a setting in the US than for those writing about research conducted in other countries. The code switching was less demanding for the contributors who had already developed an expertise in a research methodology, like community participatory research and a transformative paradigm (Mertens, 2007) where an engagement with culture is central to its core philosophical assumptions and methods. The challenge of moving the invisible to the visible is probably most pronounced for mixed method researchers planning interventions in the light of funding mandates and adapting research methods where recognizing the role of community and culture is not part of its foundational assumptions.

REFERENCES Creamer, E. G. (2018). An introduction to fully integrated mixed methods research. Sage. Creswell, J. W. (2010). Mapping the developing landscape of mixed method research. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 45–68). Sage. Gone, J. P., & Kirmayer, L. J. (2010). On the wisdom of considering culture and context in psychopathology. In T. Millon, R. F. Krueger, & E. Simonsen (Eds.), Contemporary directions in psychopathology: Scientific foundations of the DSM-V and ICD11 (pp. 72–96). Guilford Press. Hitchcock, J. H., Nastasi, B. K., Dai, D. Y., Newman, J., Jayasena, A., Bernstein-Moore, R., Sarkar, S., & Varjas, K. (2005). Illustrating a mixed-method approach for validating culturally specific constructs. Journal of School Psychology, 43, 259– 278. https://psycnet.apa.org/doi/10.1016/j. jsp.2005.04.007 hooks, b. (1984). Feminist theory: From margin to center. South End Press. Maxwell, J. A. (2018). The ‘silo problem’ in mixed methods research. International Journal of Multiple Research Approaches, 10 (1), 317–327. Mertens, D. M. (2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research, 1(3), 212225. https:// doi.org/10.1177%2F1558689807302811 Plano Clark, V. L., & Ivankova, N. V. (2016). Mixed methods research: A guide to the field. Mixed Methods Research Series. Sage. Poth, C. N. (2020). Confronting complex problems with adaptive mixed method research practices. Caribbean Journal of Mixed Methods Research, 1(1), 29–46.

27 Culturally Responsive Mixed Methods Evaluation Design Jori N. Hall and Ayesha S. Boyce

INTRODUCTION Recently, programme evaluators and mixed methods researchers have given increased attention to the role of culture and context in enquiry design. Bolstering their efforts is the evaluation and research communities’ explicit acknowledgment of the role of culture. For instance, in 2011, the American Evaluation Association (AEA) released a Public Statement on Cultural Competence in Evaluation emphasizing the importance of recognizing culture in all phases of evaluation design. The Statement defines culture, noting that it constitutes the shared experiences of people, including their languages, values, customs, beliefs, and mores. It also includes worldviews, ways of knowing, and ways of communicating. Culturally significant factors encompass, but are not limited to, race/ethnicity, religion, social class, language, disability, sexual orientation, age, and gender. Contextual dimensions such as geographic region and socioeconomic circumstances are also essential to shaping culture. (AEA, 2011)

AEA’s statement suggests that evaluation occurs within a complex set of cultural dimensions that must be acknowledged. In mixed methods

research, being attentive to culture is considered important to bring the consequences of design and their connections to issues of social justice to the fore (Mertens, 2013). Additionally, explicit attention to these issues has been taken up by over 40 professional organizations, including the American Educational Research Association (AERA, 2020). The emphasis on culture is based on concerns associated with implementing enquiry designs that generate superficial understandings, inaccurate findings or negative outcomes for participants—particularly those who are marginalized (Gordon et al., 1990; Hall, 2020). Attention to culture, then, mandates evaluators who use mixed methods designs not only advance the goals of a particular design (i.e., triangulation), but also respond to the cross-cultural dimensions of contexts, and promote equity and social justice. Furthermore, given the increase of globalization in the world (Molina-Azorin & Fetters, 2019), responding to cultural complexity and understanding how evaluation can advance culturally responsive mixed methods design is even more important; thus, our chapter provides an understanding and example of the role of cultural responsiveness when using mixed methods in evaluation. As culturally responsive mixed methods evaluators and researchers, it is important to share our

398

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

positionality and reflexivity. The first author is a Black female who brings expertise in cultural responsiveness in the areas of evaluation (Hall et al., 2020), focus group research (Hall, 2020) and mixed methods enquiry (Cooper & Hall, 2016). The second author is a Black and Latina female who has expertise in social justice-oriented (Boyce, 2019) and culturally responsive evaluation, especially within contexts with marginalized populations and science, technology, engineering and mathematics (STEM) (Boyce, 2017). To conceptualize and write this chapter, we held numerous discussions to reflect on the influence of culture (Hall, 2021) and evaluation practice on mixed methods design and innovation (Mertens & Hesse-Biber, 2013). We also reflected on the use of mixed methods in our own evaluation work; in particular, the evaluation of a STEM education evaluation project at a Historically Black College or University (HBCU) funded by the National Science Foundation (NSF). Because we both received our PhDs from predominantly White universities, we did not fully understand the HBCU context. Our lack of cultural understanding increased the importance of (a) functioning as a change agent; (b) engaging researcher reflexivity; and (c) “mixing” with intention. Our reflections concerning this HBCU cultural context enabled reflexivity—the ability to revisit our understandings about the theoretical foundation for and practice of culturally responsive mixed methods work in evaluation. This chapter presents our reexamined understandings. We first provide a theoretical foundation for our chapter, discussing the ways various fields— most notably, education and evaluation—have theorized cultural responsiveness. Following this grounding, we explicate our stance on culturally responsive mixed methods enquiry, focusing on our values-engaged, equity-focused and anti-racist commitments. Next, we highlight the researcher practices—functioning as a change agent, engaging researcher reflexivity, and “mixing” with intention—necessary to enhance responsiveness and advance equity. Last, for our practice-oriented example, we outline the cultural dynamics of the STEM context and discuss the implementation of researcher practices, focusing on how they shaped the mixed methods evaluation design.

THEORETICAL CONTRIBUTIONS TO CULTURALLY RESPONSIVE THINKING AND PRACTICE In this section, we discuss the literature that has provided a theoretical anchor for our understanding of culturally responsive thinking and practice.

This includes literature on culture-oriented pedagogy and culturally responsive evaluation. Our discussion highlights how the initial push for culturally sensitive thinking and practice particularly in service-providing fields (i.e., evaluation, social work, health, counselling psychology) was, in many ways, a response to increased diversity in society and social injustices experienced by certain cultural groups. We also note examples of how culture-oriented thinking and practice in the field of evaluation expanded to different contexts.

Culture-Oriented Theories and Pedagogies in Education In the United States, culture-oriented theories were in large part developed in response to the racial desegregation initiatives in the 1960s and 1970s. For instance, as schools across the United States became increasingly integrated, some educators and researchers sought approaches to teach students from different cultural backgrounds (Aronson & Laughter, 2016). Over time, educators committed to social justice aligned educational programmes to goals of educational equity (Ladson-Billings, 1994). By the 1990s and early 2000s, two educational theories in K-12 education made major strides in advancing cultural responsiveness in school settings (Aronson & Laughter, 2016). The first represents Gloria Ladson-Billings’s (1994) theorizing about the nature and purpose of schooling itself. Her theory promotes a culturally responsive pedagogy “designed to problematize teaching and encourage teachers to ask about the nature of the student–teacher relationship, the curriculum, schooling, and society” (p. 483). The second theory, presented by Geneva Gay (2002), posits that content knowledge is more meaningful, interesting and accessible when culturally contextualized. Her theoretical assumption is the foundation for a culturally relevant teaching approach that emphasizes “using the cultural characteristics, experiences, and perspectives of ethnically diverse students as conduits for teaching them more effectively” (p. 106). Whereas these two theories and their associated pedagogical approaches are distinct, they both inform our culturally responsive thinking because they challenge dominant cultural norms (i.e., White-centric, Western) and recognize the importance of cultural engagement as central to (a) broadening understandings about the nature of phenomena; (b) encouraging cross-cultural dialogue; and (c) stimulating actions that promote equity and social justice.

CULTURALLY RESPONSIVE MIXED METHODS EVALUATION DESIGN

Culturally Oriented Educational Evaluators Evaluators’ need to address the complexity of programme contexts and confront issues of social injustice facilitated the use of more responsive and culturally oriented mixed methods designs in educational evaluation (Hood, 2001). For instance, as early as the 1930s, African American evaluators used mixed methods designs. Qualitative methods were a particularly valuable component of their designs to assess and respond to the severity of racial injustice in segregated schools and universities as they allowed these evaluators to conduct a historical analysis of the schools, capture students’ perceptions of schools and understand their cultural backgrounds (Hood, 2001). “Mainstream” evaluation literature has largely overlooked early African American educational evaluators’ use of mixed methods to address injustices, especially racism; yet, their work evidences the historical roots of culturally responsive evaluation (Hood, 2001). A more well-known precursor to culturally responsive evaluation comes from the work of Robert E. Stake (1975). Stake’s (1975) responsive evaluation approach emphasizes using examinations of the cultural context to set boundaries for the evaluation. Like the work of early African American evaluators, Stake’s approach considers mixed methods and views qualitative data as essential to provide a more holistic understanding of programme complexities and context (Hood, 2001). Of importance are the thick qualitative descriptions, which enable evaluators to describe programme particularities and contextualize programme outcomes. Stake’s push for more description and contextualization contributed to the emphasis on culture in educational evaluation—particularly when working with communities of colour (Hood, 2001). The increased attention to culture is evident in the promotion of cultural competence in the evaluation literature (SenGupta et al., 2004). Around the same time, culturally responsive evaluation increasingly appeared in the evaluation literature (Hopson, 2009). Both cultural competence and cultural responsiveness—informed by participatory (Cousins & Chouinard, 2012; Cousins & Whitmore, 1998) and democratic traditions (House & Howe, 1999)—suggest engaging programme participants’ culture and context (Hood et  al., 2015b) and exercising evaluator reflexivity (Symonette, 2004) are essential to responsive evaluation.

Culturally Responsive Evaluation Current characterizations of culturally responsive evaluation affirm its commitment to social justice through four key imperatives: (a) welcome diverse

399

perspectives of stakeholders—especially those who have been traditionally underrepresented in evaluation (Frierson et  al., 2002); (b) adopt an anti-deficit-based approach to evaluation that views cultural diversity as a strength and vital resource (Yarbrough et  al., 2011); (c) advance culturally sensitive theories (Bowman et  al., 2015); and (d) consider active engagement with the cultural contexts of a programme as important to understand, interpret and assess evaluation findings (Askew et  al., 2012). To enact these imperatives, the field of evaluation suggests multiple strategies to bolster cultural responsiveness: (a) using an examination of the cultural context to set boundaries for the evaluation; (b) working with a diverse evaluation team; (c) identifying stakeholders who are often overlooked or marginalized; (d) developing questions that reflect a range of stakeholder issues and interest; (e) using varied data sources to centre stakeholders’ experiences; and (f) considering the cultural context when analyzing, interpreting and disseminating the data (Hood et al., 2015a). Overall, culturally responsive evaluation asserts engagement with culture through all phases of the design (Bledsoe & Donaldson, 2015) is paramount to understanding people’s perspectives and concerns, assessing programme outcomes (Askew et  al., 2012), and forging meaningful social justice efforts. While keeping its US roots in mind is important, it is equally important to note that culturally responsive evaluation continues to expand. Theoretical developments. Evidence of its growth is observed in the extension of Alkin and Christie’s (2004) Evaluation Theory Tree, in which a metaphorical tree—featuring roots and branches—is used to illustrate evaluation theories. On the Evaluation Theory Tree, the roots illustrate evaluation purposes (social accountability, epistemology and social enquiry) and the branches represent evaluation theories related to methods, use and valuing. Mertens and Wilson (2012, 2018) extended Alkin and Christie’s (2004) Evaluation Theory Tree, adding a social justice branch. The evaluation theories represented by this branch are committed to social justice and share the assumption that both knowledge and truth are situated within historical, socioeconomical and racial infrastructures of injustice, power and privilege. Because culturally responsive evaluation upholds a commitment to social justice, it has been identified as part of the social justice branch of evaluation (Mertens & Wilson, 2012, 2018). International spaces. Culturally responsive evaluation has moved from its US roots to international evaluation spaces. An example of this is the consideration of multicultural competence, or the focused attention on the role of international

400

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

law and human rights in non-Western or nondemocratic evaluation contexts (Hanberger, 2010). Additionally, evaluation associations around the world increasingly acknowledge the importance of building evaluators’ capacity to work across cultures and provide guidance on cultivating cultural competencies in evaluation practice (AAE, 2020; ANZEA, 2015; CES, 2018; EES, 2012). STEM educational evaluation. Culturally responsive enquiry has expanded to STEM educational evaluation as historically and strategically minoritized groups such as women, ethnic minorities, persons with disabilities and individuals who come from economically disadvantaged backgrounds are underrepresented in STEM fields (National Center for Education Statistics, 2009). Recently, scholars, industry leaders, community organizers and policy makers have pushed to improve STEM education and grow the number of diverse students interested in STEM majors and careers (NSF, 2016). As such, the need for evaluations, and research on evaluations, that attend meaningfully to issues of culture, race, diversity and equity has increased, especially within the evaluation of STEM educational programming (Mertens & Hopson, 2006). Over the course of the past twenty years, there has been increased utilization of culturally responsive, values-engaged and equity-focused approaches within this unique context (Boyce et al., 2019; Reid, 2020). As a result of its popularity, challenges to the culturally responsive approaches have emerged.

CRITICISMS, CONTROVERSIES AND CONSTRAINTS The use of culturally oriented approaches has long been contested as numerous issues with centring culture in evaluation theory and practice have been identified. However, one stands out. The notion of cultural competency. In the sections that follow, we review the issues with cultural competency—a term used in many service-providing fields, including evaluation, to engage the increasing cultural diversity of the people served. Then, we discuss constraints to culturally oriented enquiry, which, in our assessment, have direct implications for its social justice aspirations.

The Notion of Cultural Competency: Criticisms and Controversies Similar to the field of education, other service-providing fields (i.e., social work, health) implemented

cultural competency programmes to train professionals in response to an increasingly diverse society (Botcheva et al., 2009; Furlong & Wight, 2011). Cultural competency in these fields emphasizes professional practice or policies that enable the delivery of services in ways that align with the needs of a particular community (Pon, 2009). While this way of theorizing cultural competency addresses people’s needs, it is also highly problematic because it overlooks the oppressive conditions (i.e., whiteness/White supremacy, colonial mentalities) that uphold and maintain power differentials (Kumagai & Lypson, 2009; Pon, 2009). As a result, this version of cultural competence reflects a form of new racism; a dismissal of or a refusal to engage with the oppressive conditions that have placed many cultural groups in a marginalized social position in the first place (Pon, 2009). Another controversy is the term “cultural competency” itself. Because the word competency “may be defined as a state or quality of being adequately or well qualified, or possessing requisite or adequate knowledge or skills in a particular area”, cultural competency suggests that one can achieve or master particular skills and outcomes with relevant cultural knowledge—for example, communicating effectively with cultural groups (Kumagai & Lypson, 2009, p. 783). Researchers, scholars and practitioners in service-providing fields opposed to this orientation assert that crosscultural work is not a set of skills that can be mastered nor is that the goal (AEA, 2011; Davis et al., 2016; Kumagai & Lypson, 2009). Further, this orientation fails to emphasize the importance of learning about how one’s culture helps or hinders their work. As a result, researchers argue that cultural competency should move away from a proficiency orientation towards an approach that incorporates (a) routine assessments of one’s cultural position (power, privilege, biases); (b) critical reflection on how one’s cultural position and the context within which the enquiry is embedded interacts with the design; and (c) a willingness to adjust the design based on those assessments (Botcheva et al., 2009; Symonette, 2004). As a consequence of its US roots, the notion of cultural competency has been criticized for its lack of attention to non-Western and non-democratic cultural contexts (Hall, 2021). This criticism resulted in the advancement of the notion of multicultural competence in evaluation, which, as mentioned earlier, is attentive to international law and human rights (Hanberger, 2010). This criticism acknowledges how notions of culture and policies promoting cultural diversity vary across international multicultural spaces and has implications for the extent to which culturally responsive evaluation can be enacted globally (Hanberger, 2010).

CULTURALLY RESPONSIVE MIXED METHODS EVALUATION DESIGN

Taken together, these criticisms importantly signal: (a) how enquiry itself is embedded and implicated in larger societal structures that create and sustain power differentials; (b) how cultural responsiveness needs to consider human rights and international law; and (c) the impact of oppressive conditions on culturally responsive enquiry efforts. These controversies and criticisms are important to understand because they inform current enquiry that centres participants’ culture. For instance, contemporary culturally responsive (Hall, 2020) and equity-focused evaluation approaches (Dean-Coffey, 2018) call for a commitment to social justice and a consideration of the inequitable social structures within which enquiry is embedded. Yet, even with these commitments, constraints to culture-oriented enquiry remain.

Constraints to Culturally Responsive Enquiry Culturally responsive enquiry, despite its evolution, faces constraints that limit its potential to tackle inequities. Botcheva et  al. (2009) discuss these constraints, most notably contextual parameters that hinder an enquirer’s autonomy, limiting the range of theoretical and methodological approaches they can use to implement enquiry designs. Botcheva et  al. (2009) also note how contextual constraints differ based on the type of enquiry context. For example, in the context of evaluation, funders are a common contextual constraint. They tend to favour designs that primarily focus on programme outcomes, which, in turn, lessens the extent to which evaluators can use more processoriented or culturally anchored designs. In these cases, evaluators lack the capacity to determine design components such as evaluation questions or reporting formats. Additionally, Botcheva et  al. (2009) highlight that evaluators often lack authority to participate in the programme development process or determine the amount of time needed to implement the evaluation. These constraints potentially lead to other issues—namely, programmes being designed in ways that may not be responsive to participants’ cultural context and evaluators feeling challenged when striving to assess these programmes in culturally responsive ways. There are also constraints with the usage of culturally responsive approaches in STEM evaluation. Previous research has highlighted that attention to culture, diversity and equity can be challenged by organizational culture and underdeveloped evaluator-stakeholder professional relationships (Boyce, 2017). Other constraints in STEM evaluation include a lack of alignment of values between

401

stakeholders and evaluators, limited training in culturally responsive approaches, half-hearted implementation and the use of buzzwords without the actual employment of a culturally responsive design. In our assessment, these constraints underscore the significance of the broader social context and how it profoundly influences culture-oriented evaluation. While we fully acknowledge the critiques and constraints concerning the execution of culturally responsive enquiry, we also recognize its potential to systematically and ethically address social inequities in a particular enquiry cultural context. Accordingly, later in the chapter, we detail how cultural responsiveness was vital to bolster the mixed methods design of a STEM evaluation. Before that, we clarify our stance on culturally responsive mixed methods evaluation design.

OUR STANCE ON CULTURALLY RESPONSIVE MIXED METHODS EVALUATION DESIGN Previous literature on culturally responsive pedagogy and evaluation theory and practice (Hood et  al., 2015a) informs our stance on culturally responsive mixed methods evaluation design. Our understanding of mixed methods enquiry also shapes our stance, which proposes mixed methods research is a type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration. (Johnson et al., 2007, p. 123)

Additionally, our stance is values-engaged (Greene et  al., 2006). Values-engaged practice is influenced by a concern for STEM educational experiences and a commitment to creating opportunities to engage participants’ values and ensure decision-makers hear their perspectives (Boyce, 2017; Greene et al., 2011; Hall et al., 2012; Reid, 2020). Our values-engaged stance guides our incorporation of educative and inclusive practices in STEM and other contexts, especially those with minoritized populations. Moreover, our notion of culturally responsive mixed methods design is informed by equity-focused (Bamberger & Segone, 2011; Dean-Coffey, 2018) and anti-racist (Thomas et al., 2018) evaluation thinking. Based on these

402

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

perspectives, important assumptions guide our mixed methods designs. First, engaging participants’ values and culture, and including them in the enquiry process is essential to culturally responsive practice—particularly with participants who are most vulnerable and/or least well-served (Boyce, 2019). Second, racist, discriminatory and other complex structural, social, political and economic conditions exist in society, and these conditions systematically deny rights and opportunities to specific groups. Third, research, evaluations, programmes and interventions do not function in isolation; rather, they operate within a complex range of social conditions that have important consequences for people’s lives. Last, consistent with our previous assumptions and in alignment with culturally responsive evaluation and transformative mixed methods research (Camacho, 2020; Mertens, 2007), we believe that enquiry must be conducted in service to equity, disrupting conditions that drive discriminatory, exclusionary, oppressive or unfair practices (Hall et al., 2023).

IMPERATIVE RESEARCHER PRACTICES FOR CULTURAL ENGAGEMENT AND MIXED METHODS DESIGN DECISION-MAKING Culturally responsive mixed methods enquirers need to attend to culture and context through all phases of their enquiry design (Bledsoe & Donaldson, 2015). Being culturally responsive through design phases ultimately involves a series of choices made by the enquirer (Chenail, 2011). Thus, in this section, we take an in-depth look at the role of the enquirer by examining three practices fundamental to making design decisions in a culturally responsive manner: (a) functioning as a change agent; (b) engaging researcher reflexivity; and (c) “mixing” with intention. We contend that these practices not only enhance the responsiveness of a mixed methods design, but also address the challenges that inevitably emerge when striving to facilitate social justice or disrupt the status quo in a particular mixed methods context.

(Molina-Azorin & Fetters, 2019). This requires enquirers to take on a particular role: change agent. Functioning as a change agent means using the cultural knowledge generated from the design to produce actionable knowledge. For example, the information generated from a mixed methods design can be shared to impact educational policy, improve health-related outcomes or counter negative cultural stereotypes. To inform their design choices, change agents include participants in aspects of the design (to the extent possible) and investigate the power and privilege (Hall, 2020) inherent in the study’s context. These practices require creating spaces for meaningful interactions and dialogue (Rodriguez et  al., 2011). These practices also require a particular mindset. Specifically, change agents recognize through their interactions that participants have agency over their lives. In this way, participants themselves are also change agents. This mindset or orientation implies that participants possess cultural knowledge that can be used to “negotiate, resist and or even transform systems of dominance” (Hall, 2020, p. 55). To be clear, this orientation does not negate the existence of oppressive structures within which participants are embedded. Rather, “culturally responsive enquirers acknowledge these restrictions while also recognizing participants are capable of addressing some of the challenges (albeit in their unique way) associated with their cultural context” (Hall, 2020, p. 55). Change agents see participants’ culture as an asset and work with communities to determine and seek change within a particular context. For example, researchers who implemented a communitybased mixed methods study with African refugees were able to “identify and co-develop a culturally responsive financial empowerment intervention project aimed at increasing financial literacy, financial self-efficacy, and overall financial capability through a peer-led intervention grounded in the community’s cultural assets” (Maleku et  al., 2021). As this example illustrates, change agents pursue their design with a methodological orientation that suggests participants have agency, and understanding power dynamics in the cultural context is vital to advance meaningful participant outcomes. Furthermore, change agents believe that social enquiry can and should embody the values of a more just society (Boyce, 2019) and be designed in service of social justice.

Functioning as a Change Agent Making appropriate mixed methods design choices begins with understanding the goals of the design. In addition to exploring the enquirer’s interests, a primary goal in culturally responsive mixed methods design is to create positive changes in people’s lives

Engaging Researcher Reflexivity Functioning as a change agent while making design decisions is challenging work. This is

CULTURALLY RESPONSIVE MIXED METHODS EVALUATION DESIGN

because change agents will “traverse an everchanging landscape of cultural complexities, power differentials, social interactions, communication challenges, and ethical dilemmas” (Hall, 2020, p. 66). To manage these challenges, change agents embrace researcher reflexivity, which involves ongoing assessments of their enquiry design and adjustments based on those assessments (Lahman et  al., 2011). Fundamental to examining and adjusting the design are continuous critical reflections on the decisions made and actions taken during the research process. Multiple techniques exist to help researchers reflect on their enquiry design. Memoing and reflective journalling are among the most popular techniques. Whatever the technique, the main point of reflective practice is to record, process and examine one’s feelings and thoughts about decisions made and actions taken (Boud, 2001). Further, reflective practice can happen during any phase of the research design (before, during or after) and completed individually or collaboratively (with research team members, content experts, programme participants, for example). In particular, reflective practice helps culturally responsive mixed methods enquirers to process, assess or address: (a) cultural exchanges between the researcher and participants; (b) dilemmas as they arise; (c) the responsiveness of mixed methods design components; and (d) the consequences of the mixed methods design for participants (Janesick, 1999). Inspired by Boud’s (2011) discussion on reflective journalling, we offer some possible questions that mixed methods researchers can use to stimulate reflective practice: How do people in the cultural context view things? What are the implications of the mixed methods design for participants? What will I do if my assumptions about the mixed methods design are wrong? How am I deciding to intervene (take action or not) at this time? An especially important question for culturally responsive mixed methods enquirers includes: In what ways do my mixed methods design decisions help or hinder equity? Reflexive practice is particularly relevant in mixed methods research that examines a “transnational phenomenon” such as migration (Horvath & Latcheva, 2019, p. 128). Horvath and Latcheva (2019) discuss three areas of concern: (1) the appropriateness of research procedures (i.e., the validity of a design to integrate the complexity of migration); (2) the justification for and consequences of using certain categories or criteria (i.e., ethnicity, language skills, region of origin); and (3) the role of migration politics (i.e., the impact of policies and regulations on migration). As Horvath and Latcheva (2019) point out, reflexive practice

403

facilitates processing and addressing many political and ethical mixed methods design decisions researchers confront. Furthermore, Symonette (2004) reminds us that practising reflexivity is necessary to clarify your role as a change agent and how participants perceive your responsiveness.

“Mixing” with Intention The practice of “mixing” design elements (i.e., methods, theories, data analysis techniques) is a central feature of mixed methods enquiry. Given the various mixes possible and the types of products or insights that are generated by mixing different design elements, we believe that strong and defensible integration rests on a design that intentionally engages and maps the intended mixing purpose. In short, we believe that mixing with intention and purpose matters. Mixing with intention means describing the overall intent of the research. To accomplish this task, researchers must answer the question: For what purpose am I using both qualitative and quantitative methods? To answer this fundamental question, mixed methods scholars have offered rationales for mixing methods (Collins et  al., 2006; Creamer, 2018). Most notably, Greene et al. (1989) offered a set of purposes for mixed methods evaluation designs: complementarity, development, expansion, triangulation and initiation. Complementarity refers to mixing methods to explore different aspects of a phenomenon. Mixed methods work employed for the purpose of development refers to the use of one method to inform the development of another method. Mixing for the purpose of expansion suggests using different methods to address different enquiry questions or components. Mixing for the purpose of triangulation refers to using qualitative and quantitative methods to investigate the same phenomenon. Because contradictions or discrepancies across data sources can occur, researchers may initiate another investigation that reframes research questions or findings to generate new or different insights. The initiation purpose, then, serves to reconcile or inform unexpected findings or data paradoxes. While the purposes identified by Greene et  al. (1989) primarily focus on the mixing of qualitative and quantitative methods, we believe they are essential to refine and target appropriate designs, and support analyses and inferences. We assert the overall purpose of a study is a major driver of all other aspects of the design. We further assert that mixing different methods with intention is a critical practice for thinking better

404

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

about the nature of the “mix” in mixed methods enquiry, and specifically about what is the nature of “social justice” that is yielded by the mixed methods design (see also Chapter 4, this volume). The practice of mixing with intention positions researchers to reckon with important questions such as, “How does the mixed method design matter in the context at hand?” For example, the purpose for a mixed methods study on older Chinese persons with depression was expansion (Zeng et al., 2012). In this case, a range of standardized and validated scales were used to determine the psychological factors associated with depression, while interviews were used to expand on (confirm or reject) the quantitative findings. Yet, even with this intentional purpose, the researchers critically questioned whether the older Chinese participants would be ashamed to describe their moods and share their stories during the interviews given the stigma attached to depression in Chinese culture (Zeng et al., 2012). Ultimately, mixing with intention was beneficial because the older Chinese adults appreciated the opportunity to share their life challenges, which led to the finding that holistic health interventions are needed for older Chinese adults. Like Zeng et al. (2012), we believe that more assertive engagement with the nature of the “mix” in a design is vital to maximize its potential contributions to the enquiry context. These three practices are not only important to enhance cultural responsiveness, but also the overall character and quality of mixed methods evaluation design. Functioning as a change agent maintains vigilance on the contribution of each design decision to establish actionable, credible and meaningful findings. Researcher reflexivity facilitates ethical practice—mindfulness of one’s actions and careful implementation of design procedures with respect for others as dictated by the cultural context. Mixing with intention contributes to the purpose and outcomes of the design, which is vital to help enquirers assess the cultural appropriateness of design components and the contributions of the design.

CULTURALLY RESPONSIVE MIXED METHODS ENQUIRY IN PRACTICE: A STEM EVALUATION EXAMPLE To facilitate the application of culturally responsive mixed methods design, this section discusses the multiple cultures (e.g., disciplinary, institutional, personal) and researcher practices that shaped the design and implementation of a STEM education evaluation.

Overview of the Context To meet STEM demands and increase underrepresented minorities’ degree completion, the NSF developed a broadening participation initiative— the Historically Black Colleges and Universities Undergraduate Program (HBCU-UP). Program awards are available to “develop, implement, and study evidence-based innovative models and approaches for improving the preparation and success of HBCU undergraduate students so they may pursue STEM graduate programs” (NSF, 2017). In November 2016, the Department of Chemistry at an HBCU in the south-eastern United States was awarded an HBCU-UP. The educational goals of this HBCU-UP were to infuse polymer science content into general and physical chemistry courses, utilize a multipronged approach to improve study habits, enhance class performance, encourage undergraduates to participate in polymer research, recruit more students to STEM majors, and increase retention and success of students in chemistry courses. We were hired to conduct the external evaluation of the project.

Cultural Dimensions Important cultural dimensions in this evaluation context included the culture of the field of STEM, the institution itself, the culture the principal investigator (PI) created in her lab and the key stakeholders’ cultures. Disciplinary. Many STEM researchers strongly identify with post-positivism, are hypothesis driven and primarily utilize quantitative methods. Rigour, hard work and sacrifice are hallmarks of the field. As we previously mentioned, systematically minoritized groups in the United States have typically had a much smaller presence in STEM professional fields than their peers (Osei-Kofi & Torres, 2015). Over the past several decades, many STEM fields have witnessed a growth in participation and degrees earned by these groups, yet they remain disproportionately underrepresented (National Academies of Sciences, Engineering and Medicine, 2018). In our experience, the exclusivity of STEM fields is lauded by some and deplored by others. Institutional. HBCUs were established within the United States in the nineteenth century to provide educational opportunities to people of African descent because they were neither legally nor culturally allowed to attend existing public and private institutions of higher education. HBCUs graduate 20 per cent of all Black undergraduate students and 25 per cent of Black students with

CULTURALLY RESPONSIVE MIXED METHODS EVALUATION DESIGN

STEM degrees (Gasman & Nguyen, 2016). In our experience, alumni have fierce loyalty and pride associated with their HBCU institution. Laboratory. Over the course of the past dozen years, we have experienced a variety of laboratories and their associated cultures across institution types. Typically, the PI who runs the lab has set the tone for the lab. We have seen labs that operate much like a family, where students have a strong sense of belonging and the development of a science identity and self-efficacy are given precedence. We have also seen labs where PIs operate as if students are indentured servants whose work is neither applauded nor respected. We have interviewed students who have worked in toxic environments where they were yelled at or their research was stolen. The chemistry lab in this context had a no-nonsense but nurturing atmosphere. The PI, a Black woman scientist, was supportive but strict. Her interactions reminded us of a loving aunt who had high expectations but was committed to her students’ success. Personal. While we are both Black women in the academy, our families, upbringing, geographic location, educational pathways, personal and professional experiences distinguish us from one another. Our shared ideologies around the importance of culture, values and social justice within social enquiry situate us in similar ontological, epistemological and axiological locations.

Attending to Culture throughout the Mixed Methods Evaluation Design Process Functioning as a change agent. As change agents, we viewed our culturally responsive design as important to understand and address issues in STEM education: the problematic nature of chemistry gatekeeper courses and the attrition rates of Black students in STEM majors. Our commitment to values engagement focused our attention on the importance of preparing Black students at an HBCU to go into a field where they are not the majority. Attending to race in this context meant ensuring that our evaluation did not reflect Black students as a homologized group. In these ways, our evaluation design promoted the centring of culture, race and issues of diversity within an HBCU context which was necessary, but is rarely done in STEM evaluation contexts (Mertens & Hopson, 2006). Specifically, our design allowed for formal data collection about participant feelings of inclusion, differentiation of outcomes based on demographics and potential sustainability within

405

the Chemistry Department. For another STEM example prioritizing cultural responsibilities in mixed methods research with underrepresented communities, see also Chapter 16 (this volume). Engaging researcher reflexivity. We began this work by locating our own values, prejudices and identities (Boyce & Chouinard, 2017). To better understand this particular HBCU context and address stakeholders’ values and concerns, throughout the evaluation we (a) met and debriefed after data collection activities; (b) had informal and formal conversations with our client, the PI and student participants; and (c) took notes about the changes we made with the project based on budget constraints, timeline and data collection. For example, we utilized pre-post assessments to better understand study participants’ gains in knowledge. We had originally asked participants to take these assessments at the beginning of class. However, during one observation we noticed that students were especially concerned about these “tests” and were changing answers and talking through the pre-assessments with each other. We quickly realized that our initial assessments were no longer valid and for the next semester we placed all the assessments for the evaluation online. We also spent more time informing students about the purpose of the assessments and reminded them that their scores were not attached to their grades in the class. As a result of our reflexivity, our evaluation and conclusions increased validity and trustworthiness. “Mixing” with intention. We began this, and all our evaluation projects, by deciding what it was that we needed to investigate. Then, in consultation with our client and relevant literature, we developed research and evaluation questions. After these steps were completed, we selected appropriate data-collection methods. Quantitative measures and indicators were used to track student outcomes and multiple interviews, and observations and focus groups were used to solicit stakeholders’ voices. This mix of data allowed us to assess project goals while still ensuring independence, rigour and trustworthiness. A key purpose for this mix was triangulation, which allowed for stakeholders’ experiences to be examined multiple times and from multiple vantage points. Our intentional mixing process resulted in the inclusion of voices that might not have otherwise been heard and a nuanced understanding of the context. For example, during our observations, we noticed there were a few students who regularly came late to the class and often seemed disengaged during the laboratory exercises. Informal interviews with these students unearthed their desire to do well in class was impeded by their participation

406

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

in a university athletic team. We spoke to students about the importance of accountability and student–athlete stereotypes, and shared their experiences with the PI. These conversations and other data assisted in the restructuring of the course curriculum for its second iteration.

CONCLUSIONS Believing in the necessity of attending to the cultural context in mixed methods design, in this chapter, we shed light on the notion of cultural responsiveness. In particular, we discussed some of its theoretical contributions, key characteristics and evolution. We also considered criticisms of associated terms (cultural competency) and contextual constraints to culturally responsive enquiry. In addition, we provided our particular stance on culturally responsive mixed methods design, a set of practices essential to enhance the responsiveness and credibility of a mixed methods design, and an example to illustrate what culturally responsive mixed methods enquiry looks like in practice. In the practical example, we attended to institutional, disciplinary and personal cultures. Informal and formal conversations with the PI and Black undergraduate students were particularly helpful to engage these different cultures. As change agents, the use of mixed methods allowed us to pursue a design that allowed us to consider the different identities (i.e., student– athletes) of the Black undergraduate STEM students, thereby disrupting the view of Black undergraduates as a homologized group. Our researcher reflexivity assisted with dealing with issues that arose during the evaluation related to the pre-assessments. And our intentional utilization of mixed methods facilitated an in-depth understanding of participants’ different experiences within the Chemistry Department. While there were challenges to the implementation of the culturally responsive mixed methods design, as noted in the example, the design did facilitate changes to the course curriculum. Of course, the set of practices and the practical example we offered could never fully capture the range of cultural complexities, ethical dilemmas, joys, actions, surprises and missed opportunities that culturally responsive mixed methods enquiry embodies. Yet, what we have provided in this chapter does make important contributions; it: (a) acknowledges the cultural complexity embedded in society, a particular enquiry context and the experiences of the mixed methods enquirer;

(b) elucidates the links between mixed methods enquiry and culture, context, life experiences and the improvement of social conditions; and (c) directs attention to issues concerning structurally maintained inequities and power relations, as well as the consequences they have for conducting culturally responsive mixed methods enquiry and the people served by the enquiry. As culturally responsive Black women enquirers, we are concerned about the implications of long-standing stereotypes and injustices, particularly for cultural groups who are marginalized. As reflective, intentional change agents, we believe we must transform mixed methods enquiry in ways that will advance a more just and equitable society. Our chapter is an invitation for other Black and non-Black enquirers to consider how their mixed methods designs can do the same.

WHAT TO READ NEXT Hall, J. N. (2020). The other side of inequality: Using standpoint theories to examine the privilege of the evaluation profession and individual evaluators. American Journal of Evaluation, 41(1), 20–33. https://doi.org/10.1177/1098214019828485

Mixed methods evaluators who endeavour to be culturally responsive need to locate their own identities of privilege and oppression, and recognize how the knowledge achieved from marginalized groups can make privilege more visible. In this article, Jori N. Hall offers standpoint theories as a lens to identify privilege and advance social justice in evaluation practice. Boyce A. S. (2017). Lessons learned using a valuesengaged approach to attend to culture, diversity, and equity in a STEM program evaluation. Evaluation and Program Planning, 64, 33–43. doi: 10.1016/j.evalprogplan.2017.05.018

In this article, Ayesha S. Boyce reports on lessons learned from implementing a mixed methods design guided by the values-engaged, educative (VEE) approach to evaluate a multi-year STEM education program. Her empirical examination moves the notion of cultural responsiveness from theory to practice, providing practical guidance on implementing the VEE approach. Greene, J. C., DeStefano, L., Burgon, H., & Hall, J. N. (2006). An educative, values-engaged approach to evaluating STEM educational programs. New Directions for Evaluation, 2006(109), 53–71. https://doi.org/10.1002/ev.178

CULTURALLY RESPONSIVE MIXED METHODS EVALUATION DESIGN

While the VEE approach has evolved since its advent in the 2006 Critical Issues in STEM Evaluation New Directions for Evaluation volume, this seminal article presents, justifies, and illustrates the initial commitments of this evaluation approach, including context-specific explicit examinations of diversity and equity.

Other Articles to Read Greene, J. C., Boyce, A. S., & Ahn, J. (2011). ValueEngaged, Educative Evaluation Guidebook. Champaign, IL: University of Illinois at Urbana-Champaign. Created and produced with funds from the National Science Foundation. AEA eLibrary. Gomez, A. (2014). New developments in mixed methods with vulnerable groups. Journal of Mixed Methods Research, 8(3), 317–320. https://doi. org/10.1177/1558689814527879

REFERENCES African Association of Evaluation (AAE). (2020). African evaluation guidelines. https://afrea.org/ the-african-evaluation-guidelines/ Alkin, M. C., & Christie, C. A. (2004). Evaluation roots: Tracing theorists’ views and influences. Sage. American Educational Research Association (AERA). (2020). Diversity Training Changes Multisociety Letter. www.aera.net/Portals/38/Diversity%20 Training%20Changes%20Multisociety%20 Letter_1.pdf American Evaluation Association (AEA). (2011). American Evaluation Association public statement on cultural competence in evaluation. Retrieved from: www.eval.org Aotearoa New Zealand Evaluation Association (ANZEA). (2015). Evaluation standards for Aotearoa New Zealand. www.anzea.org.nz/app/ uploads/2019/04/ANZEA-Superu-Evaluation-standardsfinal-020415.pdf Aronson, B., & Laughter, J. (2016). The theory and practice of culturally relevant education: A synthesis of research across content areas. Review of Educational Research, 86(1), 163–206. https://doi. org/10.3102/0034654315582066 Askew, K., Beverly, M. G., & Jay, M. L. (2012). Aligning collaborative and culturally responsive evaluation approaches. Evaluation and Program Planning,

407

35(4), 552–557. https://doi.org/10.1016/j. evalprogplan.2011.12.011 Bamberger, M., & Segone, M. (2011). How to design and manage equity-focused evaluations. UNICEF. Bledsoe, K., & Donaldson, S. (2015). Culturally responsive theory-driven evaluations. In S. Hood, R. Hopson, K. Obeidat, & H. Frierson (Eds.), Continuing the journey to reposition culture and cultural context in evaluation theory and practice. Information Age. Botcheva, L., Shih, J., & Huffman, L. (2009). Emphasizing cultural competence in evaluation. American Journal of Evaluation, 30, 17–188. https://doi. org/10.1177/1098214009334363 Boud, D. (2001). Using journal writing to enhance reflective practice. New Directions for Adult and Continuing Education, 90, 9–18. https://doi. org/10.1002/ace.16 Bowman, N. R., Dodge Francis, C., & Tyndall, M. (2015). Culturally responsive indigenous evaluation: A practical approach for evaluating Indigenous projects in tribal reservation contexts. In S. Hood, R. Hopson, & H. Frierson. (Eds.), Continuing the journey to reposition culture and cultural context in evaluation theory and practice (pp. 335– 360). Information Age Publishing. Boyce, A. S. (2017). Lessons learned using a valuesengaged approach to attend to culture, diversity, and equity in a STEM program evaluation. Evaluation and Program Planning, 64, 33–43. https://doi. org/10.1016/j.evalprogplan.2017.05.018 Boyce, A. S. (2019). A re-imagining of evaluation as social justice: A discussion of the Education Justice Project. Critical Education, 10(1), 1–19. https://doi. org/10.14288/ce.v10i1.186323 Boyce, A. S., Avent, C., Adetogun, A., Servance, L., DeStefano, L., Nerem, R., & Platt, M. (2019). Implementation and evaluation of a biotechnology research experience for African-American high school students. Evaluation and Program Planning, 72, 162–169. https://doi.org/10.1016/j.evalprogplan. 2018.10.004 Boyce, A. S., & Chouinard, J. (2017). Moving beyond the buzzword: A framework for teaching culturally responsive approaches to evaluation. Canadian Journal of Program Evaluation, 32, 266–279. https://doi.org/10.3138/cjpe.31132 Camacho, S. (2020). From theory to practice: operationalizing transformative mixed methods with and for the studied population. Journal of Mixed Methods Research, 14(3), 305–335. https://doi. org/10.1177/1558689819872614 Canadian Evaluation Society (CES). (2018). Competencies for Canadian evaluation practice. Retrieved from: https://evaluationcanada.ca/txt/2competenc iescdnevaluationpractice2018.pdf

408

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Chenail, R. J. (2011). Ten steps for conceptualizing and conducting qualitative research studies in a pragmatically curious manner. The Qualitative Report, 16(6), 1715–1732. https://doi.org/ 10.46743/2160-3715/2011.1324 Collins, K., Onwuegbuzie, A., & Sutton, L., (2006). A model incorporating the rationale and purpose for conducting mixed-methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4(1), 67–100. Cooper, J., & Hall, J. N. (2016). Understanding Black male student athletes’ experiences at a historically Black university: A mixed methods approach. Journal of Mixed Methods Research, 10(1), 46–63. https://doi.org/10.1177/155868981455845 Cousins, J. B., & Chouinard, J. A. (2012). Participatory evaluation up close: An integration of research-based knowledge. Information Age Publishing, Inc. Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5–23. https://doi.org/10.1002/ev.1114 Creamer, E. G. (2018). An introduction to fully integrated mixed methods research. Sage. Davis, E. D., DeBlaere, C., Brubaker, K., Owen, J., Jordan, T. A., Hook, J. N., & Van Tongeren, D.R. (2016). Microaggressions and perceptions of cultural humility in counseling, Journal of Counseling and Development, 94, 483–493. https://doi. org/10.1002/jcad.12107 Dean-Coffey, J. (2018). What’s race got to do with it? Equity and philanthropic evaluation practice. American Journal of Evaluation, 39(4), 527–542. https://doi.org/10.1177/1098214018778533 European Evaluation Society (EES). (2012). The EES Evaluation capabilities framework. https://europeanevaluation.org/wp-content/uploads/2020/04/ EES-EVALUATION-CAPABILITIES-FRAMEWORK.pdf Frierson, H. T., Hood, S., & Hughes, G. B. (2002). Strategies that address culturally-responsive evaluation. The 2002 user-friendly handbook for project evaluation (pp. 63–73). National Science Foundation. Furlong, M., & Wight, J. (2011). Promoting “critical awareness” and critiquing “cultural competence”: Towards disrupting received professional knowledges. Australian Social Work, 64(1), 38–54. https://doi.org/10.1080/0312407X.2010. 537352 Gasman, M., & Nguyen, T-H. (2016). Historically Black colleges and universities as leaders in STEM. Philadelphia, PA: Penn Center for Minority Serving Institutions. https://cmsi.gse.rutgers.edu/sites/ default/files/MSI_HemsleyReport_final.pdf Gay, G. (2002). Preparing for culturally responsive teaching. Journal of Teacher Education, 53, 106–116. https://doi.org/10.1177/002248710205300 2003

Greene, J. C., Boyce, A. S., & Ahn, J. (2011). Valuesengaged, educative, evaluation guidebook. University of Illinois, Urbana-Champaign. Created and produced with funds from the National Science Foundation. AEA eLibrary. Greene, J. C., Caracelli, J., & Graham, W. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https://doi.org/10.3102/01623737011003255 Greene, J. C., DeStefano, L., Hall, J. N., Burgon, H., & Johnson, J. (2006). An educative, values-engaged approach to evaluation. Journal of Research Methodology, 19(2), 181–197. https://doi.org/10.1002/ ev.178 Gordon, E. W., Miller, F., & Rollock, D. (1990.) Coping with communi-centric bias in knowledge production in the social sciences. Educational Research, 19, 14–19. https://doi.org/10.3102/0013189X 019003014 Hall, J. N. (2020). The other side of inequality: Using standpoint theories to examine the privilege of the evaluation profession and individual evaluators. American Journal of Evaluation, 41(1), 20–33. https://doi.org/10.1177/1098214019828485 Hall, J. N. (2021). Cultivating cultural competence. Canadian Journal of Program Evaluation, 36(2), 191–209. https://doi.org//10.3138/cjpe.70053 Hall, J. N., Ahn, J., & Greene, J. C. (2012). Valuesengagement in evaluation: Ideas, illustrations, and implications. American Journal of Evaluation, 33(2), 195–207. https://doi.org/10.1177/10982 14011422592 Hall, J. N., Boyce, A., & Hopson, R. (Eds.) (2023). Disrupting program evaluation and mixed methods research for a more just society: The contributions of Jennifer C. Greene. Information Age Publishing, Inc. Hall, J. N., Freeman, M., & Colomer, S. E. (2020). A culturally responsive, educative evaluation approach: Success and missed opportunities. American Journal of Evaluation, 41(3), 384–403. https://doi.org/ 10.1177/1098214019885632 Hanberger, A. (2010). Multicultural awareness in evaluation: Dilemmas and challenges. Evaluation, 16(2), 177–191. https://doi.org/10.1177/1356 389010361561 Hood, S. (2001). Nobody knows my name: In praise of African American evaluators who were responsive. New Directions for Evaluation, 92, 31–43. https://doi.org/10.1002/ev.33 Hood, S., Hopson, R., & Frierson, H. (Eds.). (2015a). Continuing the journey to reposition culture and cultural context in evaluation theory and practice. Information Age Publishing, Inc. Hood, S., Hopson, R., & Kirkhart, K. (2015b). Culturally responsive evaluation: Theory, practice and future implications. In K. Newcomer, H. Harry and

CULTURALLY RESPONSIVE MIXED METHODS EVALUATION DESIGN

J. Wholey. (Eds.), Handbook of Practical Program Evaluation. (pp. 281–317). Jossey-Bass. Hopson, R. K. (2009). Reclaiming knowledge at the margins: Culturally responsive evaluation in the current evaluation moment. International Handbook of Educational Evaluation. Sage. Horvath, K., & Latcheva, R. (2019). Mixing methods in the age of migration politics: A commentary on validity and reflexivity in current migration research. Journal of Mixed Methods Research, 13(2), 127–131. https://doi.org/10.1177/1558 689818822445 House, E. R., & Howe, K. R. (1999). Values in evaluation and social research. Sage. Janesick, V. J. (1999). A journal about journal writing as a qualitative research technique: History, issues, and reflections. Qualitative Inquiry, 5(4), 505–524. https://doi.org/10.1177/107780049900500404 Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007) Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, 112–133. https://doi.org/10.1177/15586898 06298224 Kumagai, A. K., & Lypson, M. L. (2009). Beyond cultural competence: Critical consciousness, social justice, and multicultural education. Academic Medicine, 84, 782–787. http://dx.doi.org/10.1097/ ACM.0b013e3181a42398 Ladson-Billings, G. (1994). The dreamkeepers. Jossey-Bass. Lahman, M. K., Geist, M. R., Rodriguez, K. L., Graglia, P., & DeRoche, K. K. (2011). Culturally responsive relational reflexive ethics in research: The three Rs. Quality & Quantity, 45(6), 1397–1414. https://doi.org/10.1007/s11135-010-9347-3 Maleku, A., Kim, Y., Kagotho, N., & Lim, Y. (2021). Journal of Mixed Methods Research, 15(2), 212– 239. https://doi.org/10.1177/155 Mertens, D. (2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research, 1, 212–225. https://doi. org/10.1177/0002764211433797 Mertens, D. (2013). Emerging advances in mixed methods: Addressing social justice. Journal of Mixed Methods Research, 7(3), 215–218. https:// doi.org/10.1177/1558689813493994 Mertens, D., & Hesse-Biber, S. (2013). Mixed methods and credibility of evidence in evaluation. In D. M. Mertens & S. Hesse-Biber (Eds.), Mixed methods and credibility of evidence in evaluation. New Directions for Evaluation, 138, 5–13. https://doi. org/10.1002/ev.20053 Mertens, D., & Hopson, R. K. (2006). Advancing evaluation of STEM efforts through attention to diversity and culture. In D. Huffman & F. Lawrenz (Eds.), Issues and challenges facing STEM education and evaluation. New Directions for Evaluation (Vol. 109, pp. 35–51). Jossey-Bass.

409

Mertens, D., & Wilson, A. (2012). Program evaluation theory and practice. (2nd ed.). Sage. Mertens, D., & Wilson, A. (2018). Program evaluation theory and practice (3rd ed.). Sage. Molina-Azorin, J., & Fetters, M. (2019). Building a better world through mixed methods research. Journal of Mixed Methods Research, 13(3), 275–281. https://doi.org/10.1177/1558689819855864 National Academies of Sciences, Engineering, and Medicine. (2018). Indicators for monitoring undergraduate STEM education. Washington, DC: The National Academies Press. https://doi.org/10.17226/ 24943. National Center for Educational Statistics, United States. Office of Educational Research & Improvement. Center for Education Statistics. (2009). The condition of education. US Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics. National Science Foundation (NSF). (2016). Broadening participation in STEM. Arlington. National Science Foundation (NSF). (2017). Historically Black Colleges and Universities Undergraduate Program. Retrieved from: https://beta.nsf.gov/ funding/initiatives/broadening-participation Osei-Kofi, N., & Torres, L. E. (2015). College admissions viewbooks and the grammar of gender, race, and STEM. Cultural Studies of Science Education, 10(2), 527–544. https://doi.org/10.1007/s11422014-9656-2 Pon, G. (2009) Cultural competency as new racism: An ontology of forgetting. Journal of Progressive Human Services, 20, 59–71. https://doi.org/ 10.1080/10428230902871173 Reid, A. M. (2020). Applying an educative approach to engage stakeholder values in evaluations of STEM research and education programmes. Evaluation Journal of Australasia, 20(2), 103–108. https://doi.org/10.1177/1035719X20918497 Rodriguez, K., Schwartz, J., Lahman, M., & Geist, M. (2011). Culturally responsive focus groups: Reframing the research experience to focus on participants. The International Journal of Qualitative Methods, 10(4), 400–417. https://doi.org/ 10.1177/160940691101000407 SenGupta, S., Hopson, R., & Thompson-Robinson, M. (2004). Cultural competence in evaluation: An overview. New Directions for Evaluation, No. 102 (pp. 5–19). Jossey-Bass. https://doi.org/10.1002/ ev.112 Stake, R. E. (1975). To evaluate an arts program. In Stake, R. E. (Ed.) Evaluating the arts in education: A responsive approach (pp. 13–31). Merrill. Symonette, H. (2004). Walking pathways toward becoming a culturally competent evaluator: Boundaries, borderlands, and border crossings. New Directions for Evaluation, 102, 95–109. https://doi.org/10.1002/ev.118

410

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Thomas, V. G., Madison, A., Rockcliffe, F., DeLaine, K., & McDonald-Lowe, S. (2018). Racism, social programming, and evaluation: Where do we go from here? American Journal of Evaluation, 39(4), 514– 526. https://doi.org/10.1177/109821401877 2910 Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation

standards: A guide for evaluators and evaluation users (3rd ed.). Sage. Zeng, W., North, N., & Kent, B. (2012). Methodological challenges to research among Chinese: Using mixed methods to investigate older persons with depression in Macau. Journal of Mixed Methods Research, 6(3), 154–165. https://doi.org/10.1177/ 1558689811419513

28 Integrating a Four-Step Japanese Cultural Narrative Framework, Ki-Shou-Ten-Ketsu, into a Mixed Methods Study Ta i c h i H a t t a

INTRODUCTION Mixed methods research to date has largely advanced using Western perspectives. For example, Western philosophies include pragmatism (Morgan, 2007), dialectical pluralism (R. B. Johnson, 2017), critical realism (Maxwell & Mittapalli, 2010) and performative paradigms (Schoonenboom, 2019). Western ideas dominate despite commentary encouraging further engagement of global scholars in mixed methods research (Creswell, 2016; Poth et al., 2018). Today, a global perspective about mixed methods has slowly emerged in various disciplines around the world (Molina-Azorin & Fetters, 2022). Empirical studies have addressed cultural topics, such as incorporating cultural models (Collins & Dressler, 2008; Maltseva, 2016; Rinne & Fairweather, 2012), developing culture-related scales and intervention programmes (Nastasi et al., 2007; Taghipoorreyneh & de Run, 2020; Ungar & Liebenberg, 2011), and investigating the relation between culture and mental health (Arnault & Fetters, 2011; Kral et al., 2012; Zeng et al., 2012). For a further description of how indigenous cultural values inform instrument development using mixed methods research, see Chapter 14 (this volume). These studies mainly address how mixed methods studies contribute to

understanding different cultures. Conversely, in some cases, culture adds to understanding mixed methods. Studies reflect cultural perspectives integrated into mixed methods projects (Harris, 2021; Lunde et  al., 2013). Indigenous methodologies frame mixed methods studies (Chilisa & Tsheko, 2014; Harris, 2021). Acknowledging the subtle tensions between Western and non-Western cultures, discussions highlight diverse cultural perspectives brought to mixed methods studies in countries around the world (Creswell & Sinley, 2017). Recently, Asia and Oceania cultural perspectives of Asia and Pan-Pacific philosophies have appeared in the mixed methods literature. Fetters and Molina-Azorin (2019) offer a theoretical argument for the similarity of mixed methods to one of the Eastern philosophies, Yin–Yang. They analyze this Eastern philosophy in terms of ontology, epistemology, methodology and axiology (Fetters & Molina-Azorin, 2019). GoodyearSmith & 'Ofanoa (2022) present a Samoan perspective, Faafaletui, which is woven together to create new knowledge from mixed methods. Martel et  al. (2022) introduce the New Zealand Mori (te ao Mori) and the Mori metaphor He awa whiria (braided river) to describe combining the strengths of two distinct worldviews into a workable whole. For an illustrative example of

412

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

how mixed methods partnerships can learn from Kaupapa M¯aori Research Principles, see Chapter 15 (this volume). In these studies, the question arises as to how culture shapes mixed methods research, not the other way around. Further, culture can shape mixed methods by integrating it with the mixed methods design process (Fetters & MolinaAzorin, 2017; Johnson et  al., 2007). It is recognized that integration embeds within mixed methods in various ways. Researchers can integrate quantitative and qualitative research in diverse methodologists on a team, in research questions, in strands in planning, in collecting data and in analyzing data within mixed methods designs (Creswell, 2022; Fetters, 2019). Looking at integration from a broad perspective rather than a specific procedure or data level, culture can integrate with mixed methods design procedures. Although the definition of the concept of culture varies among scholars (Schwandt, 2014b), some argue that rhetoric or language is central to understanding culture (Strecker & Tyler, 2009). Language in the form of dialogues among individuals occurs through a sequence of utterances or events, as told in a story or narrative plotted in time (Schwandt, 2014c). This chapter examines the impact of a Japanese cultural narrative framework, ki-shou-ten-ketsu, on mixed methods research design. Specifically, I apply this framework for analyzing qualitative dialogue in an empirical mixed methods study. I illustrate this application using the “MORE-IC Project”, and focus on dialogue between physicians and patients for 20 cases in a mixed methods study. The study occurred in Japan and addressed the informed consent process about using chemotherapy for cancer patients (Hatta et al., 2020).

THE CULTURAL NARRATIVE FRAMEWORK I begin by defining a cultural narrative framework and then narrow the discussion to a specific Japanese cultural framework. A cultural narrative framework is a story structure in a case study for depicting a series of chronological ordered events of dialogue. The dialogue can result from participant observation and be included in the data collection and analysis of the qualitative strand in a mixed methods study. Researchers can sequence the dialogues as narrative stories (Creswell & Plano Clark, 2018). Stories or narratives are generated through many kinds of symbolic media (documents, spoken words, comics, novels, movies, TV, SNS).

The concept of narrative, also known as the narrative turn in sociology, has been evolving (Hyvärinen, 2006; Kreiswirth, 2005). The concept of narrative has expanded into psychology, education, social sciences and health sciences, and writers have debated the meaning of narrative theory and structure (Herman, 2009; Hyvärinen, 2006). For example, Herman (2009) referred (at a minimum) to narratives as temporal sequences (i.e., situations and events unfolding in time). He proposed four elements that constitute a narrative: (1) situatedness; (2) event sequencing; (3) worldmaking/world disruption; and (4) what it is like. In addition to temporal sequences, a researcher can capture multiple dialogues in an interaction between people. The dialogues can be extracted and arranged as utterances in chronological order. In many cases, if the study is about observing a dialogue and depicting its characteristics, the structuring will relate to the study’s research purpose. The researcher places the analytical units (utterances, codes, topics and themes) from the dialogue in a chronological order and their temporal sequence. Further, researchers often organize multiple dialogues into a narrative and compare the narratives. Single dialogues give way to an overall temporal sequence discussion. To discuss mixed methods for multiple dialogue comparisons, I advance a narrative framework as a meta-analysis applied to each dialogue and embed the framework in the research design. Specifically, in this chapter, it will be embedded into the qualitative analysis of a mixed methods project (although it could be embedded within a joint display for more of a fine-grained analysis) (see Fetters, 2019). It is helpful also to think about the dialogue as occurring within a case or multiple cases. Creswell and Plano Clark (2018) explain: When the end goal of a mixed methods case study design is to generate and describe a case or multiple cases, the philosophical assumption tends to be an evolving, constructivist approach. The cases evolve throughout the study. […] Theory can play into a mixed methods case study design in several ways, such as informing both the quantitative and qualitative strands of research, informing the types of cases identified through both databases, or framing the case description. Theories often provide a guiding perspective for considering the case as a complex system and integrating the different data. (p. 117)

Given that cases evolve, and theory can guide analyzing cases, a narrative framework becomes an analytical meta-framework (or theory) for case analysis. The researcher takes a holistic view of the

INTEGRATING A FOUR-STEP JAPANESE CULTURAL NARRATIVE FRAMEWORK

temporal sequences derived from a single dialogue, and explores the storylines that can be explained by each common dialogue and compares multiple dialogues for the case. In summary, the major elements in a narrative framework I will present focus on dialogue, a temporal sequence, multiple conversations and a multiple case analysis.

Japanese Cultural Narrative Framework One more element completes the narrative framework I present. The dialogue occurs within a specific cultural perspective. This cultural perspective is introduced as a storyline. In the cultural narrative framework I will present and illustrate in cases about a physician-patient dialogue around informed consent about cancer treatment (as illustrated later), I used a cultural perspective from Japan called ki-shou-ten-ketsu. Ki-shou-ten-ketsu (起承転結), originated in classical Chinese poetry, and it is an expository prose style that both Japan and Korea imported from China (Hinds, 1980, 1990). In this style, ki introduces the topic, shou develops the topic, ten forms an abrupt transition via a vaguely related point, and ketsu concludes the topic (Hinds, 1980, 1983). Hinds (1980) provided the definition of each stage: 起 ki: First, begin one’s argument. 承 shou: Next, develop that. 転 ten: At the point where this development is finished, turn the idea to a subtheme where there is a connection, but not a direct connected association (to the major theme). 結 ketsu: Last, bring all of this together and reach a conclusion. This four-part organization of a story (ki-shou-tenketsu) is an important and familiar rhetorical style to the Japanese, although it is rare or non-existent in English (Hinds, 1980, 1983). Whether this Japanese prose style is distinct to Japan has been debated. Cahill (2003) found rhetorical styles in Japanese generally not unlike those of English. Kobayashi and Rinnert (2002) found Japanese writing education moving toward a style similar to Western writing (Bankier, 2014). As discussed in Bankier (2014), however, many researchers (Donahue, 1998; Guest, 2001; Kubota, 1997, 1999) emphasize the prose style as influential in Japanese writing. In addition to ki-shou-ten-ketsu, two other Japanese cultural perspectives added to my cultural narrative framework (although these played a

413

minor, indirect role in analyzing dialogue). These were shu-ha-ri and jo-ha-kyu. Shu-ha-ri is a mentoring relationship in Japanese martial arts (budo). Shu, ha, and ri refer to observing, breaking away, and transcending, respectively. J. A. Johnson (2017) referred to shu-ha-ri as follows: The origin of the shu-ha-ri pedagogy is unclear, but it most likely developed during Japan’s Muromachi (or Ashikaga) Period (1336–1573). This three-stage learning process possesses Confucian and Buddhist educational approaches and “has been substantially shaped by Zen practice”. Japanese artisans most likely adopted the shu-ha-ri concept found in Buddhist pedagogy to explain the learning experiences that occur in Noh theater, the tea ceremony (chanoyu), or flower arranging (ikebana,), which are all traditions that predate modern martial arts. It would only be natural for Japanese martial artists to borrow terminology from one art form to explain their skill acquisition process, and Zen Buddhism and martial arts (budo) have identical learning processes. (pp. 9–10)

Jo-ha-kyu is a concept of modulation and movement applied in a wide variety of traditional Japanese arts. Roughly translated to “beginning, break, rapid”, it essentially means that all actions or efforts should begin slowly, speed up and then end swiftly. These cultural perspectives become metaphors for understanding dialogue and presenting storylines. Metaphor use is not unique to Japan or East Asia as illustrated by Ma¯ori metaphors from New Zealand used by Martel et  al. (2022). However, such metaphorical rhetoric is not argumentative like Western hypotheses or theories, nor is it discussed as a premise of research like philosophy or paradigms. Moreover, it is not easy for researchers to incorporate it into a priori developed mixed methods diagram. The narrative structure is a process of drawing inferences from dialogue. These concepts, introduced thus far—dialogue, temporal sequence, multiple dialogue analysis, metaphors, and storytelling encased within a Japanese narrative framework will be illustrated in the MORE-IC project which depicts the process of physician– patient dialogue in Japan (Hatta et al., 2020).

THE MORE-IC PROJECT The purpose of “MORE-IC (the Mixed Methods Observational Research for Informed Consent MORE-IC) was to examine the physician–patient dialogues in the setting of informed consent for

414

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

the introduction of chemotherapy in Japan (Hatta et  al., 2020). I conducted this project within the context of Japanese medical culture. It is important to first begin by discussing the cultural features of Japanese informed consent for treatment and the methodological viewpoints they provide.

Qualitative Analysis within a Japanese Context of Informed Consent Informed consent aims to protect participants in medical research and patients receiving treatment through the principle of respect for autonomy, and is generally implemented within legal, ethical procedures (Beauchamp & Childress, 1996; Council for International Organizations of the Medical Sciences, 2016; Hall et al., 2012; Office for Protection from Research Risks, 1979). In Japan, as in many countries, informed consent is a part of everyday health practice. However, its meaning in Japan must be interpreted within social and cultural contexts (Fetters, 2014; Specker & Sullivan, 2017; Ohtaki et  al., 2003). The Japanese informed consent is as follows: It is notable that this human activity emerged in the context of medical informed consent, derived from both the physician and the patient, who share an interest in realizing the best possible clinical conduct (Yanagida, 1996). (Hatta et al., 2020, p. 104)

Yanagida’s (1996) perspective on informed consent means that informed consent is not observed from procedural ethics, but rather from ethics-inpractice and micro ethics (Guillemin & Gillam, 2004; Komesaroff, 1995). Komesaroff (1995), a proponent of “micro ethics”, addresses medical ethics as follows: Medical ethics is not just about the dramatic questions that are discussed widely in the popular media or in the philosophical texts. Ethics is what happens in every interaction between every doctor and every patient. (Komesaroff, 1995)

This perspective allows the researcher to explore the phenomenon of informed consent as a subject for qualitative research and reflection (Berger, 2015; Guillemin & Gillam, 2004; Pillow, 2003). This project approached informed consent in Japan by depicting narratives from qualitative research materials, such as dialogue transcripts and field notes obtained through participatory observations.

Quantitative Analysis within a Japanese Context of Informed Consent In the 2000s, with advancements in medicine, the Japanese government began promoting transition to outpatient chemotherapy (Ministry of Health Labour & Welfare [MHLW], 2007). They also sought to improve the informed consent process promoting the expectation that patients take a greater interest in their treatment prior to informed consent consultations (MHLW, 2012). In addition, under the Japanese Cancer Control Act, the Act mandated safe cancer patients’ outpatient chemotherapy. The Act also set the expectation for physicians to build trust with patients, and their understanding of underlying illness and treatment options. As a follow-up to these Japanese declarations, I developed for the MORE-IC research project a scale to measure patients’ proactive interest in treatment. This scale measured attitudes prior to consent consultations and after building a trusting relationship with their physician. The scale was named the Motivation for Treatment Index (MTI) (Hatta et al., 2016). By using it, I assessed the strength of patients’ motivation in choosing their treatment.

Mixed Methods Objective and Procedure The general rationale for using mixed methods in the MORE-IC project was to leverage the mutual strengths and offset weaknesses of quantitative and qualitative research (Creswell, 2022; Fetters & Molina-Azorin, 2017; Rossman & Wilson, 1985; Tashakkori & Teddlie, 2009). The research project’s objective was the integration of the qualitative research aim of describing informed consent cases as narratives, and the quantitative research aim of comparing informed consent with a focus on patient motivation. This integration occurred within the Japanese context of informed consent. I stated the research objective for the mixed methods study as: [the empirical objective was] to illustrate how doctor–patient dialogues differed between patients with high and low treatment choice motivation. Using this study, our methodological objective is to illustrate a highly interactive analysis of convergent mixed methods data using inductive and deductive approaches. (Hatta et al., 2020, p. 86)

Thus, I implemented a crossover-tracks analysis in a convergent design with the qualitative and

INTEGRATING A FOUR-STEP JAPANESE CULTURAL NARRATIVE FRAMEWORK

quantitative strands intertwined and informing each other (see Figure 28.1). To collect data, I conducted a participatory observation of physician–patient dialogues. In the study, I collected qualitative and quantitative data simultaneously, including verbatim transcripts, field notes and the scores of Motivation

415

for Treatment Index (MTI) for 20 cancer patients. Descriptive statistics were used to analyze the quantitative data and patients were selected for qualitative analysis according to their MTI scores (Figure 28.1, Phase 2). The qualitative data were analyzed by content analysis (Figure. 28.1, Phase 1) and by applying the Japanese cultural

Figure 28.1  Procedural diagram of the convergent study design* Note: MTI = Motivation for Treatment Index IC = informed consent *Source: Hatta et al. (2020). Crossover mixed analysis in a convergent mixed methods design used to investigate clinical dialogues about cancer treatment in the Japanese context. Journal of Mixed Methods Research, 14(1), 84–109. doi:10.1177/1558689818792793

416

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

narrative structure of ki-shou-ten-ketsu (Figure 28.1, Phase 3). In the phase of integrative interpretation of quality and quantity data, I also formed meta-inferences based on the Japanese cultural framework (Figure 28.1, Phase 4).

Applying the Narrative Framework in the Mixed Methods Study Thus, my study employed a convergent design and applied the cultural narrative framework in the qualitative data analysis process of the design (Figure 28.1). The study involved the participant observation of 20 informed consent consultations. Of the 20 cases, 10 were dialogues between breast cancer patients and oncologists, and the remaining 10 were between lung cancer patients and oncologists. I focus on the qualitative process of this mixed methods study in this chapter discussion. Temporal sequence of dialogues. The procedure began by analyzing the temporal sequence of the dialogues, two analysts in the study conducted a content analysis of the verbatim transcripts to ascertain the perspectives of the physicians’ explanations by cancer type (Figure 28.1, Phase 1). The analysts divided the consultation content into two categories: stylized dialogues, where oncologists informed their patients of task-focused elements of medical treatment, and impromptu dialogues, where oncologists responded to the personal interests of the patient. A dialogue with highly specialized medical content, such as informed consent, is unfamiliar to many analysts. Sorting out the substantive content of informed consent through content analysis and distancing oneself from the specifics helped the analysts gain a more abstract perspective of the temporal sequence of the dialogues. High and low motivation. Based on scores on the motivation scale, the two patients with the highest motivation (MTI) scores in each cancer type were selected, and identified as high-scoring patients. Similarly, four patients were designated as the low-scoring patients (Figure 28.1, Phase 2). The two analysts reviewed the verbatim transcripts of dialogues and fieldnotes of the eight cases to determine temporal sequences, and developed the storyline for the cultural narrative framework. Changes in patient motivation in dialogues. In the procedure, the analysts separately read and analyzed the eight case transcripts in depth. They followed the chronological course of the physician–patient interaction and selected ten or eleven passages (i.e., segments of dialogues) for each case. They then read these passages jointly to understand the speakers’ attitudes (e.g.,

psychological distance and intentions). On that basis, they identified apparent changes in the patients’ motivation (whether in intention, attitude or subjective norms), and noted how many passages showed the temporal changes in dialogue. Application of the Japanese narrative framework to depict stories. In the analysts’ discussion, they applied the Japanese rhetorical–analytical framework with four passages. The ki-shou-ten-ketsu was employed as an analysis strategy to depict the narrative in the medical context of informed consent (Figure 28.1, Phase 3). The definition of each stage of ki-shou-ten-ketsu reflected the context of informed consent in Japan (Figure 28.2). A focus on the “ten” stage. The analysts selected four of the original ten passages that demonstrated features of the “ten” stage (the point where this development finishes, and the dialogue turns to a connected theme indirectly, as noted earlier). The analysts moved continuously back and forth between inductive thinking (developing concepts, categories and relations from the text) and deductive thinking (testing the concepts, categories and relations against the text). They identified specific topics from each passage and organized the passages into ki-shou-ten-ketsu style. They read the lines again, trimmed the transcripts of all unnecessary content (utterances they deemed unimportant or uninformative). They discussed whether the passage indicated temporal changes in the informed consent consultation (Hatta. et  al., 2020). We focused on the tenth stage because this stage represented a turning point in the consultation in several cases, a “sudden change of quality that plays the part of a forerunner or prerequisite to slow structural change in psychometric treatment” (Bohm, 1992). This perspective allowed us to draw metainferences about the turn (ten) stage. For example, it was at the tenth stage that one patient began to talk about himself, at which point the oncologist seized the opportunity to help the patient express his real interest (turn stage for case No. 12). This patient had seemed to be a “difficult patient” until disclosing his own wish at this stage. The function of the turn in his case thus emerged as an “opportunity for the patient to become open about his real interest”. This type of function was observed in other cases as well (Nos. 24 and 29), regardless of the strength of the patient’s motivation. Highly motivated patients seemed willing to accept that they were suffering from cancer after the oncologists informed them of the seriousness of their condition (Nos. 14, 19 and 32). This was an attempt by the oncologists to direct the patients to seek agency (Schwandt, 2014a). Based on this interpretation, the function of the turn (ten) stage in these cases could be expressed

INTEGRATING A FOUR-STEP JAPANESE CULTURAL NARRATIVE FRAMEWORK

417

Figure 28.2  ki-shyou-ten-ketsu style* and the four-stage framework in clinical dialogue** * Source: Hinds, J. (1980). Japanese expository prose. Paper in Linguistics, 13(1), 117–158. doi:10.1080/08351818009370494 ** Source: Hatta et al. (2020). Crossover mixed analysis in a convergent mixed methods design used to investigate clinical dialogues about cancer treatment in the Japanese context. Journal of Mixed Methods Research, 14(1), 84–109. doi:10.1177/1558689818792793

as “the patient coming to terms with their cancer”. Conversely, we identified a third, corresponding function: “the atmosphere became tense, but there was no turn”. Patients did not open their hearts to their oncologists (Cases No. 9 and 13) during the limited period of the informed consent consultation. While the oncologists tried to approach the patients, they often found themselves struggling to elicit the patients’ agency. (Hatta et al., 2020). Applying the narrative framework for drawing inferences. Then, using the framework in the integrative inference phase (Figure 28.1, Phase 4), the analysts depicted the patient’s interest in the temporal sequence of the dialogue. Comparing the temporal sequences of the dialogues with the quantitative data allowed drawing meta-inference on the “ten” stage.

SUMMARY Scholars in the mixed methods field discuss integration occurring in multiple phases of a research project (Fetters, 2019). One approach being given increased attention is the integration of a cultural

perspective into the design of a mixed methods study. This chapter advances a cultural narrative framework from Japan, ki-shou-ten-ketsu, for analyzing qualitative dialogue in a mixed methods study. It presents analyzing dialogue for the qualitative strand of a mixed methods study focused on multiple dialogue segments. To construct the narrative, the Japanese narrative framework shaped it and focused on the turning point in the story as a key development.

Implications This study provided a broad sense of integration linked into mixed methods designs. A cultural framework integrated into the qualitative analysis and design provides a novel application of integration. Further, the discussion departs from traditional views of mixed methods procedures, such as designs, conceptual frameworks and philosophy. It introduces more “holistic” thinking into mixed methods. For example, Nisbett (2003) points out that there are epistemological differences between Westerners and East Asians that can generate very different organizations of

418

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

knowledge, even for observations of the same phenomena. According to Nisbett (2003), Western analytic thinking with humans can be understood in terms of straightforward rules, and he addresses it as follows: Westerners have a strong interest in categorization, which helps them to know what rules to apply to the objects in question, and formal logic plays a role in problem solving. East Asians, in contrast, attend to objects in their broad context. The world seems more complex to Asians than to Westerners, and understanding events always requires consideration of host factors that operate in relation to one another in no simple, deterministic way. Formal logic plays little role in problem solving. (Nisbett, 2003, p. xvi)

Thus, the Japanese perspective adopted in this study focused on building a narrative story from multiple discourses between physicians and patients. Looking for a turning point represents a metaphorical analysis of qualitative data within a four-step framework popular to Japanese. Although I find discussions about Asian philosophy useful in understanding mixed methods (the philosophy of Yin–Yang mentioned by Fetters and Molina-Azorin, 2019), this chapter took a cultural focus on mixed methods procedures. This chapter was limited to the dialogue related to informed consent in one country. Certainly, cultural links to mixed methods procedures in other countries may take a different shape. Also, this discussion focused on “ten”, the narrative turn in a dialogue rather than a discussion about all four stages of ki-shou-ten-ketsu. This discussion is drawn from the health sciences which may present a different dialogue from one involving non-health settings. Future working definitions and applications of the framework should include surveys conducted in non-East Asian cultures and studies that analyze dialogues outside of clinical contexts. Finally, other meta-frameworks may be useful for integration into mixed methods procedures.

perspectives that have dominated the mixed methods literature. Finally, it introduces a Japanese cultural perspective into mixed methods and qualitative research, and acknowledges a popular prose structure found in Japan in hopes that other cultural studies will use a narrative cultural framework and link culture into mixed methods design.

WHAT TO READ NEXT Hatta, T., Narita, K., Yanagihara, K., Ishiguro, H., Murayama, T., & Yokode, M. (2020). Crossover mixed analysis in a convergent mixed methods design used to investigate clinical dialogues about cancer treatment in the Japanese context. Journal of Mixed Methods Research, 14(1), 84–109. https://doi.org/10.1177/1558689818792793

Hatta et  al. conducted participatory observations of physician–patient dialogues prior to initiating cancer chemotherapy taking place in the sociocultural context of Japan. Readers of this article may find it thought-provoking to know why the authors had to employ mixed methods, how they managed to apply mixed methods design to the research and what their meta-inference illuminated. Fetters, M. D., & Molina-Azorin, J. F. (2019). A call for expanding philosophical perspectives to create a more “worldly” field of mixed methods: The example of Yinyang philosophy. Journal of Mixed Methods Research, 13(1), 15–18. https://doi. org/10.1177/1558689818816886

In this article, Fetters and Molina-Azorin highlighted the significant aspects of Yin-Yang philosophy as an Asian worldview, which supports the embrace of mixed methods research. The authors also emphasized the importance of scholars becoming more aware of their unique cultural traditions and thinking styles so that they can start bringing a wide range of worldviews to the mixed methods research communities.

Contribution to the Field of Mixed Methods Research

Goodyear-Smith, F., & ‘Ofanoa, M. (2022). Fa’afaletui: A Pacific research framework. Journal of Mixed Methods Research, 16(1), 34–46. https://doi. org/10.1177/1558689820985948

This chapter recommends the inclusion of diverse cultural perspectives in contrast to Western perspectives that dominate the mixed methods discussions. It further encourages thinking about integration as more than integrating databases (Creswell, 2022), and taking a broad view of integration. It moves beyond describing cultural practices and highlights philosophical Asian

In this paper, through a Samoan framework, Fa’afaletui meaning ‘ways of weaving together’, Goodyear-Smith and ‘Ofanoa have shown how this framework, derived from Pacific philosophy of connectiveness and a collective holistic approach, can contribute to create new knowledge in the field of mixed methods research. By adopting the rhetoric and metaphor found in Fa’afaletui,

INTEGRATING A FOUR-STEP JAPANESE CULTURAL NARRATIVE FRAMEWORK

the authors attempted to demonstrate how other cultural worldviews can stand as a framework of mixed methods research. To help readers understand that, the authors provided a case example of how the framework was directing their research. Martel, R., Shepherd, M., & Goodyear-Smith, F. (2022). He awa whiria—A “braided river”: An Indigenous Ma¯ori approach to mixed methods research. Journal of Mixed Methods Research, 16(1), 17–33. https://doi.org/10.1177/1558689820984028

This article attempts to present how it is appropriate and relevant for mixed methods research to turn to non-Eurocentric world views and indigenous philosophies. In their bicultural research, Martel et  al. use the M¯aori metaphor He awa whiria (braided river) to describe combining the strengths of two distinct worldviews into a “workable whole”. The authors also illustrate how a bicultural research framework can integrate different paradigms and mixed methodology by bringing out and combining their strengths when conducting bicultural research.

ACKNOWLEDGEMENTS I would like to express my sincere gratitude to Dr. John W. Creswell for his insightful suggestions on the broader context of mixed methods research and his tremendous editing help. Without his constructive imputes and support, this chapter would not have addressed the issues of culture in mixed methods research. Likewise, I am grateful to Dr. Elizabeth G. Creamer for her guidance and persistent help as a Section Editor in this Handbook. Furthermore, I would like to extend my sincere thanks to Keiichi Narita, PhD, and Hidenori Kashihara, MPH, for enormous discussions and support while undertaking the MORE-IC project.

REFERENCES Arnault, D. S., & Fetters, M. D. (2011). RO1 funding for mixed methods research: Lessons learned from the “Mixed-Method Analysis of Japanese Depression” project. Journal of Mixed Methods Research, 5(4), 309–329. https://doi.org/10.1177/1558689811 416481 Bankier, J. (2014). Experiences of Japanese writing instruction: Beliefs about rhetorical organization. In N. Sonda & A. Krause (Eds.), JALT2013 conference proceedings (pp. 509–520). JALT.

419

Beauchamp, T. L., & Childress, J. F. (1996). Principles of biomedical ethics (5th ed.). Oxford University Press. Berger, R. (2015). Now I see it, now I don’t: Researcher’s position and reflexivity in qualitative research. Qualitative Research, 15(2), 219–234. https://doi. org/10.1177/1468794112468475 Bohm, T. (1992). Turning points and change in psychoanalysis. International Journal of Psycho-Analysis, 73(4), 675-684. www.scopus.com/inward/record. url?eid=2-s2.0-0027078173&partnerID=40&md5=a fae8f3f41266d76bb2ade614092b992 Cahill, D. (2003). The myth of the “turn” in contrastive rhetoric. Written Communication, 20(2), 170–194. https://doi.org/10.1177/0741088303020002003 Chilisa, B., & Tsheko, G. N. (2014). Mixed methods in Indigenous research: Building relationships for sustainable intervention outcomes. Journal of Mixed Methods Research, 8(3), 222-233. https:// doi.org/10.1177/1558689814527878 Collins, C. C., & Dressler, W. W. (2008). Cultural consensus and cultural diversity: A mixed methods investigation of human service providers’ models of domestic violence. Journal of Mixed Methods Research, 2(4), 362–387. https://doi.org/10.1177/ 1558689808322766 Council for International Organizations of Medical Sciences. (2016). International ethical guidelines for health-related research involving humans. Switzerland: Council for International Organizations of Medical Sciences. https://cioms.ch/wp-content/ uploads/2017/01/WEB-CIOMS-EthicalGuidelines.pdf Creswell, J. W. (2016). Reflections on the MMIRA The Future of Mixed Methods Task Force Report. Journal of Mixed Methods Research, 10(3), 215–219. https://doi.org/10.1177/1558689816650298 Creswell, J. W. (2022). A concise introduction to mixed methods research (2nd ed.). Sage. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Creswell, J. W., & Sinley, R. C. (2017). Developing a culturally-specific mixed methods approach to global research. Kolner Zeitschrift Fur Soziologie Und Sozialpsychologie, 69, 87–105. https://doi. org/10.1007/s11577-017-0453-2 Donahue, R. T. (1998). Japanese culture and communication: Critical cultural analysis. University Press of America. Fetters, M. D. (2014). Bioethics and medico-legal issues in Japan. In J. D. Babb (Ed.), The SAGE handbook of modern Japanese studies (pp. 320– 333). Sage. Fetters, M. D. (2019). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Sage. Fetters, M. D., & Molina-Azorin, J. F. (2017). The Journal of Mixed Methods Research starts a new

420

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

decade: The mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11(3), 291–307. https://doi. org/10.1177/1558689817714066 Fetters, M. D., & Molina-Azorin, J. F. (2019). A call for expanding philosophical perspectives to create a more “worldly” field of mixed methods: The example of Yinyang Philosophy. Journal of Mixed Methods Research, 13(1), 15–18. https://doi. org/10.1177/1558689818816886 Goodyear-Smith, F., & ‘Ofanoa, M. (2022). Fa’afaletui: A Pacific research framework. Journal of Mixed Methods Research, 16(1), 34–46. https://doi. org/10.1177/1558689820985948 Guest, M. (2001). Culture research: A hotbed of myths, folk wisdom, and stereotypes? In D. McMurray & M. Swanson (Eds.), JALT2001 conference proceedings (pp. 602–620). JALT. Guillemin, M., & Gillam, L. (2004). Ethics, reflexivity, and “ethically important moments” in research. Qualitative Inquiry, 10(2), 261–280. https://doi. org/10.1177/1077800403262360 Hall, D. E., Prochazka, A. V., & Fink, A. S. (2012). Informed consent for clinical treatment. Canadian Medical Association Journal, 184(5), 533. https:// doi.org/10.1503/cmaj.112120 Harris, J. (2021). Mixed methods research in developing country contexts: Lessons from field research in six countries across Africa and the Caribbean. Journal of Mixed Methods Research, 16(2), 165–182. https://doi.org/10.1177/15586898211032825 Hatta, T., Narita, K., Yanagihara, K., Ishiguro, H., Murayama, T., & Yokode, M. (2016). Measuring motivation for medical treatment: Confirming the factor structure of the Achievement Motivation Index for Medical Treatment (AMI-MeT). BMC Medical Informatics and Decision Making, 16(1), 22. https://doi.org/10.1186/s12911-016-0260-0 Hatta., T., Narita., K., Yanagihara., K., Ishiguro., H., Murayama., T., & Yokode., M. (2020). Crossover mixed analysis in a convergent mixed methods design used to investigate clinical dialogues about cancer treatment in the Japanese context. Journal of Mixed Methods Research, 14(1), 84–109. https://doi.org/10.1177/1558689818792793 Herman, D. (2009). Basic elements of narrative. Wiley-Blackwell. https://doi.org/https://doi.org/ 10.1002/9781444305920 Hinds, J. (1980). Japanese expository prose. Research on Language & Social Interaction, 13(1), 117–158. Hinds, J. (1983). Contrastive rhetoric: Japanese and English. Text-Interdisciplinary Journal for the Study of Discourse, 3(2), 183–196. Hinds, J. (1990). Inductive, deductive, quasi-inductive: Expository writing in Japanese, Korean, Chinese, and Thai. In U. Connor & A. Johns (Eds.), Coherence in writing: Research and pedagogical

perspectives (pp. 87–109). Teachers of English to Speakers of Other Languages. Hyvärinen, M. (2006). Towards a conceptual history of narrative. In M. Hyvärinen, A. Korhonen, & J. Mykkänen (Eds.), The travelling concept of narrative (Vol. 1, pp. 20–41). Tutkijakollegium. https:// helda.helsinki.fi/handle/10138/25742 Johnson, J. A. (2017). From technique to way: An investigation into taekwondo’s pedagogical process. Journal of Martial Arts Anthropology, 17(4), 3–13. https://doi.org/10.14589/ido.17.4.2 Johnson, R. B. (2017). Dialectical pluralism: A metaparadigm whose time has come. Journal of Mixed Methods Research, 11(2), 156–173. https://doi. org/10.1177/1558689815607692 Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/1558689 806298224 Kobayashi, H., & Rinnert, C. (2002). High school student perceptions of first language literacy instruction: Implications for second language writing. Journal of Second Language Writing, 11(2), 91–116. https://doi.org/10.1016/S1060-3743(02) 00067-X Komesaroff, P. (1995). From bioethics to microethics: Ethical debate and clinical medicine. In P. A. Komesaroff (Ed.), Troubled bodies: Critical perspectives on postmodernism, medical ethics, and the body (pp. 62–86). Duke University Press. Kral, M. J., Links, P. S., & Bergmans, Y. (2012). Suicide studies and the need for mixed methods research. Journal of Mixed Methods Research, 6(3), 236– 249. https://doi.org/10.1177/1558689811423914 Kreiswirth, M. (2005). Narrative turn in the humanities. In D. Herman, M. Jahn, & M.-L. Ryan (Eds.), Routledge encyclopedia of narrative theory (pp. 377–382). Routledge. Kubota, R. (1997). A reevaluation of the uniqueness of Japanese written discourse: Implications for contrastive rhetoric. Written Communication, 14(4), 460–480. https://doi.org/10.1177/0741088 397014004002 Kubota, R. (1999). Japanese culture constructed by discourses: Implications for applied linguistics research and ELT. TESOL Quarterly, 33(1), 9–35. https://doi.org/https://doi.org/10.2307/3588189 Lunde, Å., Heggen, K., & Strand, R. (2013). Knowledge and power: Exploring unproductive interplay between quantitative and qualitative researchers. Journal of Mixed Methods Research, 7(2), 197– 210. https://doi.org/10.1177/1558689812471087 Maltseva, K. (2016). Using correspondence analysis of scales as part of mixed methods design to access cultural models in ethnographic fieldwork: Prosocial cooperation in Sweden. Journal of Mixed

INTEGRATING A FOUR-STEP JAPANESE CULTURAL NARRATIVE FRAMEWORK

Methods Research, 10(1), 82–111. https://doi. org/10.1177/1558689814525262 Martel, R., Shepherd, M., & Goodyear-Smith, F. (2022). He awa whiria—a “braided river”: An Indigenous Ma¯ori approach to mixed methods research. Journal of Mixed Methods Research, 16(1), 17–33. https://doi.org/10.1177/1558689 820984028 Maxwell, J. A., & Mittapalli, K. (2010). Realism as a stance for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social & behavioral research (2 ed., pp. 145167). Sage. https://doi.org/10.4135/9781506335193 Ministry of Health, Labour and Welfare. (2007). Basic plan to promote cancer control programs. Tokyo, Japan: Ministry of Health, Labour and Welfare. http://ganjoho.ncc.go.jp/data/public/statistics/ backnumber/odjrh3000000o8is- att/cancer.pdf Ministry of Health, Labour and Welfare. (2012). Basic plan to promote cancer control programs. Tokyo, Japan: Ministry of Health, Labour and Welfare. http://www.mhlw.go.jp/bunya/kenkou/dl/gan_ keikaku02.pdf Molina-Azorin, J. F., & Fetters, M. D. (2022). Books on mixed methods research: A window on the growth in number and diversity. Journal of Mixed Methods Research, 16(1), 8–16. https://doi. org/10.1177/15586898211068208 Morgan, D. L. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48–76. https://doi.org/10.1177/2345678906292462 Nastasi, B. K., Hitchcock, J., Sarkar, S., Burkholder, G., Varjas, K., & Jayasena, A. (2007). Mixed methods in intervention research: Theory to adaptation. Journal of Mixed Methods Research, 1(2), 164– 182. https://doi.org/10.1177/1558689806298181 Nisbett, R. E. (2003). The geography of thought: How Asians and Westerners think differently … and why. Free Press. Ohtaki, S., Ohtaki, T., & Fetters, M. D. (2003). Doctor–patient communication: A comparison of the USA and Japan. Family Practice, 20(3), 276–282. Pillow, W. (2003). Confession, catharsis, or cure? Rethinking the uses of reflexivity as methodological power in qualitative research. International Journal of Qualitative Studies in Education, 16(2), 175–196. https://doi.org/10.1080/0951839032000060635 Poth, C. N., Fetters, M. D., & Molina-Azorin, J. F. (2018). Distinct yet synergetic contributors to mixed methods research: Intersections for MMIRA and JMMR. Journal of Mixed Methods Research, 12(1), 3–10. https://doi.org/10.1177/1558689817743581 Office for Protection from Research Risks, National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (1979). The Belmont report: Ethical principles and

421

guidelines for the protection of human subjects of research. Department of Health, Education, and Welfare, National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. Rinne, T., & Fairweather, J. (2012). A mixed methods approach: Using cultural modeling and consensus analysis to better understand New Zealand’s international innovation performance. Journal of Mixed Methods Research, 6(3), 166–183. https://doi. org/10.1177/1558689811420572 Rossman, G. B., & Wilson, B. L. (1985). Numbers and words: Combining quantitative and qualitative methods in a single large-scale evaluation study. Evaluation Review, 9(5), 627–643. https://doi.org/ 10.1177/0193841X8500900505 Schoonenboom, J. (2019). A performative paradigm for mixed methods research. Journal of Mixed Methods Research, 13(3), 284–300. https://doi. org/10.1177/1558689817722889 Schwandt, T. A. (2014a). Agency. In The SAGE dictionary of qualitative inquiry. https://books.google. co.jp/books?id=OEgoBgAAQBAJ Schwandt, T. A. (2014b). Culture. In The SAGE dictionary of qualitative inquiry. https://books.google. co.jp/books?id=OEgoBgAAQBAJ Schwandt, T. A. (2014c). Narrative. In The SAGE dictionary of qualitative inquiry. https://books.google. co.jp/books?id=OEgoBgAAQBAJ Specker Sullivan, L. (2017). Dynamic axes of informed consent in Japan. Social Science & Medicine, 174, 159-168. https://doi.org/https://doi.org/10.1016/ j.socscimed.2016.12.031 Strecker, I., & Tyler, S. (2009). Culture and Rhetoric (I. Strecker & S. Tyler, Eds.). Berghahn Books. Taghipoorreyneh, M., & de Run, E. C. (2020). Using mixed methods research as a tool for developing an Indigenous cultural values instrument in Malaysia. Journal of Mixed Methods Research, 14(3), 403424. https://doi.org/10.1177/1558689819857530 Tashakkori, A., & Teddlie, C. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage. Ungar, M., & Liebenberg, L. (2011). Assessing resilience across cultures using mixed methods: Construction of the child and youth resilience measure. Journal of Mixed Methods Research, 5(2), 126–149. https://doi.org/10.1177/155868 9811400607 Yanagida, K. (1996). Inspiring informed consent. Chuohoki. Zeng, W., North, N., & Kent, B. (2012). Methodological challenges to research among Chinese: Using mixed methods to investigate older persons with depression in Macau. Journal of Mixed Methods Research, 6(3), 154–165. https://doi.org/10.1177/ 1558689811419513

29 Leveraging Mixed Methods Community-based Participatory Research (MMCBPR) in Diverse Social and Cultural Contexts to Advance Health Equity P. P a u l C h a n d a n a b h u m m a , A n n i k a A g n i and Melissa DeJonckheere

INTRODUCTION Community-based participatory research (CBPR) is a research approach used in health and other relevant settings in which academic researchers, community members and other stakeholders work collaboratively and equitably in all aspects of the research process to address relevant issues that impact community well-being and health equity (Israel et  al., 2013; Wallerstein et  al., 2018; W. K. Kellogg Foundation, 2001). We use the term “community” to refer to a group of individuals with commonly shared interests, identities, norms or needs (Israel et  al., 2018), which may include individuals with shared geographic locations, cultural beliefs or traditions, professions, health statuses or other characteristics. In CBPR, community partners contribute to key aspects of the research process, including conceptualization of a community-oriented research problem, data collection and analysis, and dissemination of findings to diverse audiences. By building on the strengths and priorities of community partners, CBPR is often considered a

more equitable, sustainable and culturally relevant framework for conducting collaborative research to achieve health equity and social justice (Wallerstein et  al., 2018). CBPR could be potentially combined with many qualitative and quantitative approaches reflecting the diverse perspectives that partners bring to the research partnership (Creswell & Plano Clark, 2018). Mixed methods researchers have increasingly embraced the integration of core mixed methods designs with another methodology to address complex research contexts (Creswell & Plano Clark, 2018; Plano Clark & Ivankova, 2016). Accordingly, one could “thread” participatory elements throughout the mixed methods research (MMR) procedures (e.g., research conceptualization, data collection, analysis, interpretation and dissemination) and core design (e.g., convergent, explanatory and exploratory sequential, etc.) (Creswell & Plano Clark, 2018). For an illustrative example, Chapter 19 in this volume discusses the use of case study mixed methods and mixed methods case study designs. A typology of advanced MMR design, known as mixed methods community-based participatory research (MMCBPR),

LEVERAGING MIXED METHODS COMMUNITY-BASED PARTICIPATORY RESEARCH

423

Table 29.1  Key terms and definitions Mixed methods research (MMR) Community-based Participatory Research (CBPR)

Mixed Methods Community-based Participatory Research (MMCBPR) Culture

Intentional mixing of quantitative and qualitative approaches in a single study (or series of studies) A collaborative approach to actively and equitably engage community members in all phases of the research process, often oriented toward improving the local community’s health and advancing health equity (Israel et al., 2013; Wallerstein et al., 2018). Intersection/integration of mixed methods approaches within a community-based participatory research framework (DeJonckheere et al., 2019). An internalized and shared framework consisting of dynamic and interconnected ecological elements (and adapted to resources and constraints in the ecologic system) that is used by group (or subgroup) members to ensure their survival and as well as to provide individual and collective meaning for and in life (Kagawa-Singer et al., 1993, 2016; Hartigan, 2010).

Source: Author Created

emerges when researchers apply core mixed methods designs within a CBPR framework (DeJonckheere et  al., 2019). We use the term “MMCBPR” to capture the research approach that integrates elements of MMR and CBPR: the use of qualitative and quantitative approaches in data collection, analysis and interpretations in MMR in conjunction with the engagement with community partners in CBPR to promote equitable partnerships and realize social justice and health equity. In this chapter, we advance a compelling argument and provide guiding examples for integrating CBPR with MMR as a distinctive model for promoting health equity in diverse social settings and cultural contexts of academic-community partnerships. The purpose of the chapter is to illustrate the added value of embedding CBPR within MMR in socially and culturally diverse communities as a strategy for mixed methods researchers to respond to community needs, address health disparities and promote community health and well-being. In the sections that follow, we begin by disclosing our individual backgrounds and their influence on our research approach. Next, we provide a brief overview of health disparities in the US and outline the practices of CBPR to advance health equity. We describe MMCBPR, provide examples of its implementation across diverse social and cultural contexts in the US, and describe challenges and opportunities for stakeholders conducting MMCBPR. Our focus on the US reflects the setting of our research, as well as the social and cultural contexts with which we are most familiar. However, we also discuss the applications and transferability of MMCBPR design to global contexts. We conclude with a discussion of the ways in which MMCBPR can be further advanced. Table 29.1 defines key terms in this chapter.

Researcher Background and Approach We believe in the importance of addressing positionality in this chapter. Here, we disclose the ways in which our multicultural identities, experiences, training and worldview shape how we approach research and make sense of our findings (Lincoln & Guba, 1985). I (PPC) am an LGBT-identified Thai scholar with research and training experience in community-engaged, mixed methods, decolonization and health equity research. Since my early public health training, I became interested in understanding mechanistically the contributions of CBPR to health and social equity. Now as a mixed methods researcher, I have developed an appreciation for the synergistic alignment between MMR and CBPR. Integrating different ways of knowing through MMR can enhance a community-engaged research approach that unites academic and community perspectives to promote health equity. In recent editorial and professional capacities, I have also advocated for meaningful approaches to promote health equity in MMR and public health practices (Fetters et  al., 2020; Narasimhan & Chandanabhumma, 2021). I (AA) am a female Indian-American undergraduate student at the University of Michigan– Ann Arbor. I am currently a research assistant in a MMR study investigating experiences of hypoglycemia in adults with type 1 diabetes, and a scoping review of MMR articles involving racial and ethnic disparities across the cancer care continuum, among other projects. I am interested in learning more about MMCBPR within the health science and public health fields in particular. I (MD) am a White American woman with doctoral training in educational and communitybased action research. I was first introduced to CBPR and MMR in my graduate programme, and

424

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

both transformed the way I perceived my place in research. I aim to approach research from a collaborative and participatory lens that elevates the experiences of individuals and communities. In practice, this means that research questions and study designs are developed based on the preferences of the focal population rather than my interests as an academic researcher. As an adolescent health researcher, community-engaged research approaches are particularly valuable to me to help ensure that the study design, interpretation and dissemination efforts are consistent with the current needs of young people. In recent years, I have been working to advance the use of CBPR in the mixed methods field so that the impact of MMCBPR can be better understood and evaluated (DeJonckheere et al., 2019; Jones et al., 2020).

Leveraging Mixed Methods Community-based Participatory Research to Enhance Health Equity in the United States Brief overview of health disparities in the United States

In the US, health disparities (i.e., potential avoidable differences in health in which less socially advantaged groups are systematically at worse health disadvantage than their counterparts) are evident across communities of diverse social backgrounds and cultural contexts due to structural racism, homophobia, xenophobia, ableism, among other forms of social oppression (Braveman, 2006). For example, relative to non-Hispanic White individuals, People of Colour experience disproportionate exposure to and adverse outcomes from cardiovascular diseases, cancer and other major health concerns (Williams, 2012; Yonas et  al., 2006). Disproportionate rates of hospitalization, mortality and other health consequences of the COVID-19 pandemic have shed light on the persistence of health inequities and their underlying structural determinants (Chowkwanyun & Reed, 2020; Ford, 2020). Although “communities” have often been portrayed as the focal recipients of health disparities research in the US, the degree of community involvement and participation in research practices has varied considerably. Scholars and practitioners across health and social science disciplines have increasingly advocated for collaborative research partnerships with impacted communities to address health disparities (Cacari-Stone et  al., 2014; Israel et  al., 1998; Wallerstein & Duran, 2010).

Community-based participatory research to advance health equity

CBPR is a form of community-engaged research, a tradition of research approaches that typically involve collaboration with community members as active participants (rather than as passive study participants) to address issues that shape their wellbeing (McCloskey et  al., 2011). Variants of community-engaged research approaches include participatory action research, community-partnered participatory research, tribal participatory research and empowerment evaluation, among other approaches (Israel et al., 2013; Jones & Wells, 2007; Wallerstein et al., 2018). These variants may differ in their purpose, specific process and members involved in the process, but commonly involve equitable community participation in key aspects of these approaches. In this chapter, we focus on CBPR and its intersection with MMR (i.e., MMCBPR) because it is a popular approach in US health research and offers potential transferable applications to the global context. CBPR has arisen in response to demands for democratizing scientific research from communities that have historically experienced marginalization and abuse from public health research and practice, resulting in research mistrust and inequities (Little 2009; Mercer and Green 2008; Wallerstein & Duran 2006; Wallerstein et  al., 2019). The past 30 years have seen greater advocacy in the US for the use of CBPR, as reflected in increased funding, training and dissemination efforts devoted to CBPR (Mercer & Green, 2008; Viswanathan et  al., 2004). Systematic reviews and meta-analysis of CBPR and related approaches suggest their beneficial impacts on health and social outcomes (Cyril et  al., 2015; O’Mara-Eves et  al., 2015; Questa et  al., 2020; Salimi et al., 2012). Israel and colleagues (2013, 2018) drew upon the literature and their field expertise to synthesize the following current key principles of CBPR: (1) recognition of community as a unit of identity; (2) building on strengths and resources within the community; (3) facilitation of collaborative and equitable partnership in all phases of research and power-sharing processes that address social inequities; (4) promotion of co-learning and capacity building among all partners; (5) achieving a balance of research and action for the mutual benefit of all partners; (6) emphasizing locally relevant public health problems and ecological perspectives; (7) involving cyclical and iterative process of systems development; (8) participatory dissemination of findings and knowledge to all partners; (9) long-term process and commitment to sustainability; and

LEVERAGING MIXED METHODS COMMUNITY-BASED PARTICIPATORY RESEARCH

(10) addressing issues of race, ethnicity, racism and social class, and embracing cultural humility. They recognized that these evolving principles should be adapted to the unique context, purpose and participants of each partnership, and that not all principles will be achieved in all partnerships. CBPR is likely to involve community partners who differ in race, ethnicity, gender orientation, education attainment, socioeconomic position and disability status, among other dimensions of membership differences (Chandanabhumma et al., 2022; Israel et al., 2018). As a result, CBPR partnerships should embrace cultural humility, or commitment to self-critique and reflection, address power imbalances, and establish and maintain authentic partnership (Tervalon & Murray-Garcia, 1998). CBPR researchers are encouraged to consider how structural inequities contribute to emergent cultural differences within partnerships, and work towards remediating power imbalances. Further examination of how research designs that employ CBPR, including MMCBPR, navigate potential social and cultural differences between the research and community counterparts to navigate different worldviews, as well as to balance knowledge and action being generated, is needed. In addressing the culture of research partnerships, we recognize the influence of multiple cultural domains in the realm of research, although the very concept of culture is often inadequately examined in health research (Dressler et al., 2005). There is often insufficient elaboration of the role of culture in shaping the nature and conduct of health research in the US (Kagawa-Singer et  al., 2016). We also recognize that culture does not necessarily equate to the community culture, but could encompass any of these domains that can distinctively and variably influence the conduct of research: the culture of the community of focus, the culture of science, the culture of the researcher and the larger societal culture (Kagawa-Singer et al., 2016). As an illustrative example, Chapter 27 (this volume) examines the influence of multiple cultural domains in transformative evaluation research and highlights culturally responsive mixed methods practice. Members of these cultural domains should be considered in their historical, geographic, social and political contexts and how these factors influence their positions in the societal hierarchy, again often inadequately accounted for when research identifies so-called “issues” of diverse cultural groups (KagawaSinger et  al., 2016). For this chapter, we centre the culture of the focal partnership communities in order to illustrate the ways in which research partners navigate social and cultural contexts to advance health equity using MMCBPR.

425

Intersection of Community-based Participatory Research and Mixed Methods Research CBPR frameworks may be particularly impactful for mixed methods researchers who seek to prioritize community needs, build on the strengths and resources of a community, and promote equitable research practices. Through these communitydriven principles, CBPR has been demonstrated to enhance the quality, relevance and sustainability of research initiatives across the health disciplines (Balazs & Morello-Frosch, 2013). It follows, then, that MMCBPR may similarly offer opportunities for investigators to promote health transformations through a horizontal research model that prioritizes partnership with community members rather than a top-down, expert-driven model. MMCBPR has proliferated in the past few years in response to the call for intersecting multiple theoretical and methodological paradigms to address the “wicked problems” of health inequities and meaningfully advance health equity (Curry & Nunez-Smith, 2015; Mertens, 2015). In a recent methodological review of MMCBPR studies, DeJonckheere et  al. (2019) identified 129 articles that addressed a range of topics in the health and social sciences (e.g., nutrition, sexual health, behavioural health, cancer, environmental health, education, etc.). Their review found that MMR was most often used during the conduct of original research, as opposed to other phases of CBPR such as partnership formation or partnership evaluation. In the reviewed articles, authors described choosing a CBPR framework to develop more culturally relevant and sustainable interventions, increase participation of the focal population, collect local knowledge and expertise, and determine the focus of research. Their approach was later replicated in another methodological review of 29 MMCBPR studies of mental health (Jones et al., 2020). While DeJonckheere et al. (2019) and Jones et al. (2020) have detailed the methodological features of studies using a MMCBPR design, description of how the research partnerships responded to the social and cultural contexts influencing their studies was lacking. Further elaboration on how research partnerships using MMCBPR make decisions about conducting their research is essential to understanding how to implement MMCBPR and leverage the benefits of this approach.

Examples of MMCBPR in the US

In the following section, we review recent examples of published MMCBPR studies to explore the ways in which research partnerships used

426

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

MMCBPR principles to navigate diverse social and cultural contexts of their partnership settings. We selected MMCBPR articles in the US published since the DeJonckheere et  al. (2019) methodological review (i.e., after 2017). Through these examples, we share how the authors have articulated their use of MMCBPR to conduct research that responds to the distinctive contexts (e.g., culture, language, identities, socioeconomic status, historical experiences, etc.) of their partnership communities to advance health equity research. Each example includes an academic-community partnership that prioritized the needs and preferences of the focal community to improve the impact of their research. The studies include mixed methods approaches to develop and test culturally tailored interventions (e.g., Corvin et  al., 2017; Grant et  al., 2021; McCarley et al., 2021; McKinley et al., 2019) or evaluate the effectiveness of their partnerships (e.g., Zoellner et al., 2021). McKinley et  al. (2019) described their use of a convergent mixed methods design implemented through a CBPR partnership. Their collaborative and culturally relevant approach was grounded in a theoretical model (Framework of Historical Oppression, Resilience and Transcendence) to help develop a substance abuse and violence prevention intervention that was responsive to the needs and experiences of American Indian (AI) and Alaskan Native (AN) communities. This study took place within the context of oppression that has negatively impacted AI/AN communities, and the continued disproportionate rates of mental, behavioural and physical health problems faced. As raised by the authors, interventions for AI/ AN communities have not always been responsive to the needs of the focal community and have often caused harm to the community as a result. Therefore, McKinley et al. aimed to determine culturally specific risk and protective factors related to substance abuse and violence that could positively impact their community. Inclusion of protective factors is a key consideration of MMCBPR studies that align with CBPR principles, such as building on community strengths. In addition, the authors emphasized the importance of engaging tribal members in data collection and analysis. In their study, tribal members contributed to both the quantitative (survey) and qualitative (field notes, participant observations, interviews) data collection and analysis. Involving tribal members as key members of the research team helped to ensure cultural sensitivity, allow for perspectives that are unique to the AI/AN community and may otherwise be dismissed by non-Native perspectives and contribute to a decolonizing process. Each of these strategies is aligned with the CBPR principles of

equitable partnership and power sharing, co-learning and capacity building, and addressing systems of oppression while embracing cultural humility (Israel et  al., 2018). Additionally, the authors noted the need for researchers to practice reflexivity, to give back to the community, and to follow tribal protocols while also acknowledging that historical oppression (i.e., colonialism) continues to affect this community. Grant et  al. (2021) used MMCBPR to test a school-based intervention to improve physical activity during recess among children on an American Indian reservation using a sequential mixed methods design. The authors described the unique social and cultural context of collaborating with AI communities on a reservation, including their efforts to collaborate with and foster trust with tribal leaders and a regional tribal review board. The authors also described the importance of using a CBPR approach to help alleviate mistrust and support resilience in their partnership community. The design included focus groups to develop strategies to increase physical activity, and the resulting intervention was tested quantitatively. Throughout their study, the research team communicated with community members to solicit feedback, seek approval, and co-disseminate findings. They shared social and cultural considerations that influenced their study context and described how eight of Israel et  al.’s (2018) CBPR principles were applied, including leveraging the resources of the community, equitable collaboration and power-sharing between academic and community partners, colearning and capacity building, and participatory dissemination of findings. To help foster trust, results were shared with community members throughout the study, allowing community leaders to understand their community’s needs and allowing the research team to integrate community perspectives into the research. Furthermore, authority was assigned to regional tribal IRB as opposed to the university IRB, as the regional tribal IRB was aware of CBPR approaches and was familiar with tribal knowledge. The incorporation of CBPR in their mixed methods design allowed the research team to build trust, honour resources and knowledge of the community, respond to the needs of the community and counteract the impact of historical trauma. Corvin et al. (2017) detailed their CBPR partnership in a multiphase mixed methods translational study that focused on obstacles in chronic disease self-management programmes among Latinos experiencing both chronic disease and minor depression. The authors described the specific social and cultural contexts that underlaid their research: Latinos in the US have

LEVERAGING MIXED METHODS COMMUNITY-BASED PARTICIPATORY RESEARCH

disproportionately high rates of chronic illness and depression, while also facing inequitable gaps in healthcare access and quality. Needed were evidence-based interventions that respond to barriers to care for Latinos with co-occurring chronic conditions and depression, including access to healthcare, lack of resources or services and linguistic or cultural differences. In their study, the authors first conducted focus groups and interviews with individuals with chronic disease along with family members, community healthcare workers and leaders of community organizations to design and then quantitatively test the feasibility of a selfmanagement programme. Based on the authors’ description of their research partnership, several key CBPR principles are evident. First, the study aimed to address health disparities that were specific to the local Latino population (i.e., attending to issues of race and ethnicity). Second, the authors included the perspectives and knowledge of multiple stakeholder groups in the development of an evidence-based intervention that was responsive to the needs and resources of the local community. For example, findings from the qualitative phase and discussions with the community advisory board indicated that “depression” may carry a negative association for their focal population. To avoid this stigma, similar words that do not directly translate to “depression” were instead used. Collaborative attention to language and cultural perspectives may influence the rigour of research and implementation of interventions. As in previous examples, the authors found that participants emphasized the importance of powersharing and trust building. Participants advised that the programme should be implemented with a trusted community member, and that the best approach would be to engage with a bilingual community member who shared lived experiences with chronic disease and depression. Finally, the authors described the CBPR principles of disseminating findings to diverse audiences and promoting both research and action, whereby study findings were translated into real-world practice to help reduce health disparities experienced in the local community. McCarley et  al. (2021) elaborated on their CBPR partnership in their multiphase explanatory sequential mixed methods study that helped improve an intervention to address dietary disparities in a diverse community of Latinos. The authors described the importance of using an MMCBPR approach to navigate the social and cultural contexts that influenced their research. The local community comprised a diverse group of Latinos, who were predominantly immigrants from Central America. The intervention encouraged Latino parents of young children

427

to substitute sugar-sweetened beverages with tap water, because sugar-sweetened beverage consumption was high within this community, while plain water consumption was low. With input from the community advisory board during bilingual meetings, a bilingual curriculum intervention was delivered during home visits to provide information about the safety and health benefits of US tap water, as well as tips designed to increase water consumption in a community where mistrust of tap water was common. Their study highlights several important CBPR principles that were integrated into their MMCBPR design, including addressing an issue of local importance, valuing local resources and expertise, co-learning among academic and community partners, disseminating findings to improve the local community and sustaining their partnership over time (over 12 years at the time this research was conducted). The intervention was evaluated using quantitative assessments (baseline and follow-up surveys) and in-depth qualitative interviews conducted after the intervention was complete. The data were integrated through discussion between academic partners and a childcare organization, with summarized survey findings, qualitative themes and illustrative quotes presented together. Finally, Zoellner et  al. (2021) conducted a convergent, parallel mixed methods case study to investigate the impact of a Community-Academic Advisory Board (CAB) in the context of comprehensive cancer control. The CAB consisted of both community and academic members, including healthcare workers, cancer survivors and university staff. Several unique social and cultural factors informed their MMCBPR study design. For example, cancer is a leading cause of death in rural communities, and disparities in rural populations include lack of early screening and detection in addition to limited treatment and support after diagnosis. In their research programme, MMCBPR was used to elevate local knowledge to develop a culturally relevant intervention. Interviews with CAB members resulted in several recommendations to improve comprehensive cancer control and CAB functioning; notably, enhanced internal and external communication and the addition of new CAB members for additional perspectives. Other significant findings from interviews demonstrated the value of collaboration, inclusive decisions and clear role assignments, as well as the need for improved accountability processes and more effective use of the assets of CAB members towards the research plans. Their MMCBPR and research findings highlight the importance of several CBPR principles in their work, including equitable collaboration and partnership, power-sharing

428

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

and balancing research and action to evaluate and improve the effectiveness of the CAB.

Opportunities and challenges for implementing MMCBPR design

In alignment with the principles and values of CBPR articulated by Israel et  al. (2013, 2018) and described earlier, the partnerships above utilized MMCBPR to optimize their research and increase the impact of their work. In the examples above, we see common strategies used to address the specific social and cultural characteristics of their research: 1 Integrating conceptual frameworks and theoretical perspectives that acknowledge past and present forces of oppression (e.g., Framework of Historical Oppression, Resilience and Transcendence; Decolonial Framework). 2 Seeking culturally nuanced understanding of community assets and constraints to increase the cultural relevance of the partnership research or intervention. 3 Establishing trust between partners and historically oppressed/marginalized communities. 4 Attending to language and cultural perspectives unique to a community that influence participation and interpretation in research. 5 Ensuring inclusive and equitable decisionmaking to incorporate both community and academic partners’ perspectives. Importantly, multiple overlapping strategies from the list above are often woven throughout the course of the studies to respond to the specific social and cultural contexts of the research partnership. Despite the potential of community-engaged research approaches to advance health equity, mixed methods researchers may encounter the following challenges: • Application of community-engaged research principles. While the MMCBPR literature offers grounding principles to shape partnerships and research, not all principles will be applicable in all contexts. Each partnership should dedicate time and efforts to collectively determine what principles are important for their project and what goals to strive for at that time. Although they recognize that certain core values (e.g., equitable collaboration) underlie many partnerships, Israel et al. (2018) emphasize that the key principles of CBPR should not be prescribed to partnerships

“as is.” Also, community as a unit of identity may undermine heterogeneity among community members and the complexity of power relationships between academic and community partners (Wallerstein et al., 2019). • Evolving research questions. When research topics and research questions are communitydriven, the focus of research can change over time. It can be difficult to formulate a research team with broad subject matter expertise. In many of the examples above, the research teams consulted regularly with community partners to address the community concerns and ensure they were meeting the needs of the community. However, mixed methods research teams with experience adapting to evolving research questions (when using quantitative results to drive a qualitative phase, for example) may be more comfortable with flexibility and responsiveness. Indeed, Poth (2020) details the importance of using adaptive practices, which allow for unforeseen changes in a study to be addressed as they evolve, when employing mixed methods research approaches to undertake complex social issues. • Adapting to changing environments. Changing environments also influence the sustainability of MMCBPR projects. For example, challenges related to the COVID-19 pandemic significantly impacted the development and implementation of research partnerships. It can be helpful to leverage partnership strengths, values and resources to respond to these changes. In addition, it is important to agree upon the long-term goals and vision of the partnership, including how success will be monitored over time. Once there is shared understanding of success and sustainability, research teams can engage in activities to promote the sustainability of partnership activities or of relationships among partners (Chandanabhumma et  al., in press). Proactive planning and discussion about sustainability is preferred to last-minute discussions at the close of a project or funding period. • Power sharing and decision making in diverse contexts. Although CBPR emphasizes equitable partnership throughout the research process, the degree to which community partners contribute to each specific phase of the research process can in practice vary depending on each study context (Israel et al., 2018). In some of the examples presented, community members served in

LEVERAGING MIXED METHODS COMMUNITY-BASED PARTICIPATORY RESEARCH

advisory roles that helped to guide the conduct of research, while in others, they contributed to data collection and analysis procedures. However, it is important to engage in power sharing and collaborative practices that acknowledge and affirm the cultural diversity of all partners, including developing mutual shared language (i.e., of research and community practice), building trust, fostering respect, and cultivating appreciation for various cultures and differences in goals, interests, and norms of CBPR engagement (Israel et al., 2018). For instance, Chapter 16 in this volume provides further discussion on prioritizing cultural responsiveness in mixed methods research engagement with underrepresented communities. Deliberate practices to address inequitable relationships, such as open dialogue on inequities, attention to research hierarchies, honouring power sharing and community-driven outcomes, are encouraged to fully realize equity within and beyond partnerships (Wallerstein et al., 2019).

Applicability and Transferability of MMCBPR Design to Global Contexts This chapter showcases illustrative applications of MMCBPR that were conducted in the US, but application is certainly not limited to this location. CBPR concepts are historically rooted in movements from across the world and continue to be applied globally to address health inequities. CBPR approaches have been leveraged globally (e.g., Kamanda et  al., 2013; Pinto et  al., 2012), and the intersection of MMR in CBPR frameworks is increasingly common (e.g., GoodyearSmith et  al., 2016; Murray et  al., 2013). In the section that follows, we describe both the historical roots and contemporary applications of MMCBPR outside of the US. Importantly, CBPR is historically grounded in intellectual movements in the Global North (i.e., the Northern traditions within North America and Europe) and the Global South (i.e., the Southern traditions within Latin America, Asia and Africa). For example, the emphasis on equitable decision-making among community members in CBPR draws from Kurt Lewin’s conceptualization of action research, an approach to research that emphasizes the translation of theory into action through iterative problem-solving cycles in collaboration with practitioners in organizations (Lewin, 1997). From the Global South/Southern Tradition, Paolo Freire’s emphasis on the liberation of knowledge led to

429

CBPR principles that empower community members to be experts of their own lived experiences (Freire, 1970). Additionally, feminist, post-structural and decolonizing theories that have developed globally provide additional perspective on the nature of research (Wallerstein & Duran, 2018). At present, there are emerging variants of CBPR indicative of adaptations to specific contexts across the globe. For example, Ibhakewanlan and McGrath (2015) proposed African Communitybased Research (ACBR) Methodology in which the incorporation of African perspectives renders the approach more suitable for the partnership context and experiences in Africa. The principles of CBPR explicitly incorporate the intimate relationship between the individual and the community, the connection between individual knowledge and social knowledge, and the respect for the surrounding environment. The emergence of these practices suggests that MMCBPR has been applied to and should continue to be promoted across various global contexts in accordance with equitable approaches that decentralize Western-oriented perspectives and meaningfully honor the wisdom of global communities (Chandanabhumma & Narasimhan, 2020). MMCBPR has been used outside the US to identify priorities for health research among community members in Uganda (Dowhaniuk et al., 2021), determine community needs and preferences for health programmes in Japan (Haya et  al., 2020), and describe the nature of CBPR partnerships with rural Aboriginal communities (Snijder et al., 2020), among many other health equity research topics. These scholars have also described the importance of acknowledging historical trauma and oppression, partnering with community members to enhance research and promoting culturally specific data-collection techniques (see, for example, research protocol from Munro et al., 2017). While the specific community needs, strengths and resources will vary in each context, a similar process can be used to engage in MMCBPR in any setting: develop partnerships over a dedicated period of time, establish mechanisms to promote trust, identify the unique needs of the community from their own perspective and advance power-sharing and collaborative decision-making practices.

Advancing the Intersection of Community-based Participatory Research and Mixed Methods Research As health inequities remain a challenge in the US and globally, MMCBPR offers an opportunity to

430

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

build partnerships that will improve the relevance, sustainability and quality of research. MMCBPR teams can respond to the specific circumstances of their research setting, topic and partnership through attention to their guiding theoretical perspectives, community strengths and resources, trust (and mistrust), language and cultural barriers, and inclusive and equitable decision-making. To further advance the use of MMCBPR and achieve health equity, it is important for future research to examine best practices in MMCBPR, capture variation and applications in global contexts, and document community-oriented and capacity outcomes of MMCBPR. With recent increases in the use, support and training mechanisms for both community-engaged research and mixed methods research approaches in federally funded research (Coyle et al., 2018; Viswanathan et al., 2004), we expect to see a continued proliferation of community-engaged mixed methods research that continues to evolve in response to the social and cultural complexities of an increasingly globalizing world in the coming years. Future research could explore the unique contributions of MMCBPR to health equity across the globe.

ACKNOWLEDGEMENTS The authors gratefully acknowledge the assistance of Rania Ajilat in preparing this chapter manuscript.

WHAT TO READ NEXT Chapter 27 (this volume)

In this chapter, Hall and Boyce examine the role of culture and consider the impact of values embedded in academic disciplines in their transformative evaluation research. The authors attend to the wider socio-cultural influence of professional guidelines in defining culture, closely examine the intersections between identity and principles of the scientific culture in science and engineering disciplines, and employ critical race theory to inform their analysis. Chandanabhumma, P. P., & Narasimhan, S. (2020). Towards health equity and social justice: An applied framework of decolonization in health promotion. Health Promotion International, 35(4), 831–840. https://doi.org/10.1093/heapro/ daz053

There is a need to address the colonial legacy in global health promotion activities. In this article, the authors proposed a literature-informed framework that applies the process of decolonization into health promotion practice. The framework may help practitioners attend to colonizing structures in health promotion and engage with communities to advance their visions of health equity and social justice. DeJonckheere, M., Lindquist-Grantz, R., Toraman, S., Haddad, K., & Vaughn, L. M. (2019). Intersection of mixed methods and community-based participatory research: A methodological review. Journal of Mixed Methods Research, 13(4), 481–502. https://doi.org/10.1177/1558689818778469

In a methodological review, DeJonckheere et. al describe the use of MMCBPR in 129 peerreviewed publications across fields. The review characterizes the current state of MMCBPR in the literature, including key methodological features, and provides recommendations for improving reporting of MMCBPR to further advance the framework.

REFERENCES Badiee, M., Wang, S. C., & Creswell, J. W. (2012). Designing community-based mixed methods research. In D. K. Nagata, L. Kohn-Wood, & L. A. Suzuki (Eds.), Qualitative strategies for ethnocultural research (pp. 41–59). American Psychological Association. https://doi.org/10.1037/13742-003 Balazs, C. L., & Morello-Frosch, R. (2013). The three Rs: How community-based participatory research strengthens the rigor, relevance, and reach of science. Environmental Justice, 6(1), 9–16. Braveman, P. (2006). Health disparities and health equity: Concepts and measurement. Annual Review of Public Health, 27, 167–194. https://doi.org/ 10.1146/annurev.publhealth.27.021405.102103 Cacari-Stone, L., Wallerstein, N., Garcia, A. P., & Minkler, M. (2014). The promise of community-based participatory research for health equity: A conceptual model for bridging evidence with policy. American Journal of Public Health, 104(9), 1615– 1623. https://doi.org/10.2105/AJPH.2014.301961 Chandanabhumma, P.P., Fàbregues, S., Oetzel, J., Duran, B., & Ford, C. (2022). Examining the influence of group diversity on the functioning of community-based participatory research participants: A mixed methods study. American Journal of Community Psychology, 1–13. https://doi.org/ 10.1002/ajcp.12626.

LEVERAGING MIXED METHODS COMMUNITY-BASED PARTICIPATORY RESEARCH

Chandanabhumma, P. P., Gabrysiak, A., Brush, B. L., Coombe, C. M., Eng, E., Jensen, M., Lachance, L., Shepard, P., Wallerstein, N. B., & Israel, B. A. (in press). Cultivating an ecosystem: A qualitative exploration of sustainability in long-standing community-based participatory research (CBPR) partnerships. Progress in Community Health Partnerships: Research, Education, and Action. Chandanabhumma, P. P., & Narasimhan, S. (2020). Towards health equity and social justice: An applied framework of decolonization in health promotion. Health Promotion International, 35(4), 831–840. https://doi.org/10.1093/heapro/daz053 Chowkwanyun, M., & Reed, A. L. (2020). Racial health disparities and Covid-19 – caution and context. The New England Journal of Medicine, 383, 201–203. https://doi.org/10.1056/NEJMp2012910 Corvin, J., Aguado Loi, C., Alfonso, M., Martinez Tyson, D., Chan, I., Maria, P., & Gonzales, J. (2017). Translating research into practice: Employing community-based mixed methods approaches to address chronic disease and depression among Latinos. The Journal of Behavioral Health Services & Research, 44(4), 574–589. https://doi.org/10.1007/ s11414-016-9525-8 Coyle, C. E., Schulman-Green, D., Feder, S., Toraman, S., Prust, M. L., Plano Clark, V. L., & Curry, L. (2018). Federal funding for mixed methods research in the health sciences in the United States: Recent trends. Journal of Mixed Methods Research, 12(3), 305– 324. https://doi.org/10.1177/1558689816662578 Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Curry, L. A., & Nunez-Smith, M. (2015). Mixed methods in health sciences research: A practical primer. Sage. Cyril, S., Smith, B. J., Possamai-Inesedy, A., & Renzaho, A. M. N. (2015). Exploring the role of community engagement in improving the health of disadvantaged populations: A systematic review. Global Health Action, 18(8), 29842. https://doi. org/10.3402/gha.v8.29842 DeJonckheere, M., Lindquist-Grantz, R., Toraman, S., Haddad, K., & Vaughn, L. M. (2019). Intersection of mixed methods and community-based participatory research: A methodological review. Journal of Mixed Methods Research, 13(4), 481–502. https://doi.org/10.1177/1558689818778469 Dowhaniuk, N., Ojok, S., & McKune, S. L. (2021). Setting a research agenda to improve community health: An inclusive mixed-methods approach in Northern Uganda. PLoS One, 16(1), e0244249. https://doi.org/10.1371/journal.pone.0244249. Dressler, W. W., Oths, K. S., & Gravlee, C. C. (2005). Race and ethnicity in public health research: Models to explain health disparities. Annual

431

Review of Anthropology, 34(1), 231. https://doi. org/10.1146/annurev.anthro.34.081804.120505 Fetters, M.D., Wu, J.P., & Chandanabhumma, P.P. (2020). Words matter: Calling the community of research to recognize, react, and remove racializing research rhetoric. Journal of Mixed Methods Research. https://doi.org/10.1177/15586898 20977233 Ford, C. L. (2020). Addressing inequities in the era of COVID-19: The pandemic and the urgent need for critical race theory. Family and Community Health, 43(3), 184–186. https://doi.org/10.1097/FCH. 0000000000000266 Freire, P. (1970). Pedagogy of the oppressed. Seabury Press. Goodyear-Smith, F., Corter, A., & Suh, H. (2016). Electronic screening for lifestyle issues and mental health in youth: A community-based participatory research approach. BMC Medical Informatics and Decision Making, 16(1), 1–8. https://doi. org/10.1186/s12911-016-0379-z Grant, V., Brown, B., & Harris, K. J. (2021). Lessons learned in conducting community participatory school-based health research on an American Indian Reservation. Progress in Community Health Partnerships, 15(1), 75–83. https://doi.org/ 10.1353/cpr.2021.0007 Haya, M. A. N., Ichikawa, S., Shibagaki, Y., Akame Machijuu Genki Project Community Advisory Board, Wakabayashi, H., & Takemura, Y. (2020). The “Healthy Akame!” community – government – university collaboration for health: A community-based participatory mixed-method approach to address health issue in rural Japan. BMC Health Services Research, 20(1), 1100. https://doi.org/10.1186/ s12913-020-05916-w Ibhakewanlan, J-O., & McGrath, S. (2015). Toward an African community-based research (ACBR) methodology. Sage Open. https://doi.org/10.1177/ 2158244015613106 Israel, B. A., Eng, E., Schulz, A. J., Becker, A. B., Allen, A. J., Guzman, J. R., & Lichtenstein, R. (2018). Critical issues in developing and following CBPR principles. In N. B. Wallerstein, B. M. Duran, J. G. Oetzel, & M. Minkler (Eds.), Community-based participatory research for health (pp. 31–44). Jossey-Bass. Israel, B. A., Eng, E., Schulz, A. J., & Parker, E. A. (2013). Introduction to methods in communitybased participatory research for health (2nd ed.). In B. A. Israel, E. Eng, A. J. Schulz, & E. A. Parker (Eds.), Methods in community-based participatory research for health (pp. 1–42). Jossey-Bass. Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health,

432

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

19(1), 173–202. https://doi.org/10.1146/annurev. publhealth.19.1.173 Jones, D. E., Lindquist-Grantz, R., & DeJonckheere, M. (2020). A review of mixed methods community-based participatory research applications in mental health. Journal of Social, Behavioral, and Health Sciences, 14(1), 254–288. Jones, L., & Wells, K. (2007). Strategies for academic and clinician engagement in community-participatory partnered research. Journal of the American Medical Association, 297(4), 407–410. https://doi. org/10.1001/jama.297.4.40 Kagawa-Singer, M. (1993). Redefining health: Living with cancer. Social Science & Medicine, 37(3), 295–304. https://doi.org/10.1016/0277-9536 (93)90261-2 Kagawa-Singer, M., Dressler, W., & George, S. (2016). Culture: The missing link in health research. Social Science & Medicine, 170, 237–246. Kamanda, A., Embleton, L., Ayuku, D., Atwoli, L., Gisore, P., Ayaya, S., Vreeman, R., & Braitstein, P. (2013). Harnessing the power of the grassroots to conduct public health research in sub-Saharan Africa: A case study from western Kenya in the adaptation of community-based participatory research (CBPR) approaches. BMC Public Health, 13(1), 1–10. https://doi.org/10.1186/1471-245813-91 Lewin, K. (1997). Resolving social conflicts and field theory in social science. American Psychological Association. https://doi.org/10.1037/10269-000 Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Sage. Little, P. C. (2009). Negotiating community engagement and science in the Federal Environmental Public Health Sector. Medical Anthropology Quarterly, 23(2), 94–118. https://doi.org/10.1111/ j.1548-1387.2009.01049.x McCarley, S., López-Ríos, M., Burgos Gil, R., Turner, M. M., Cleary, S. D., Edberg, M., & Colón-Ramos, U. (2021). Using a community-based participatory mixed methods research approach to develop, evaluate, and refine a nutrition intervention to replace sugary drinks with filtered tap water among predominantly Central-American immigrant families with infants and toddlers: The Water up@ Home Pilot Evaluation study. Nutrients, 13(9), 2942. https://doi.org/10.3390/nu13092942 McCloskey, D., Aguilar-Gaxiola, S., & Michener, J. (2011). CTSA Community Engagement Key Function Committee Task Force on the Principles of Community Engagement. Center for Disease Control (CDC). www.atsdr.cdc.gov/communityengagement/pce_ctsa.html McKinley, C. E., Figley, C. R., Woodward, S. M., Liddell, J. L., Billiot, S., Comby, N., & Sanders, S. (2019). Community-engaged and culturally relevant research to develop behavioral health

interventions with American Indians and Alaska natives. National Center for the American Indian and Alaska Native Mental Health Research (US), 26(3), 79–103. https://doi.org/10.5820/aian.2603. 2019.79 Mercer, S. L., & Green L. W. (2008). Federal funding and support for participatory research in public health and health care. In M. Minkler & N. Wallerstein (Eds.), Community-based participatory research for health (pp. 399–406). Jossey-Bass. Mertens, D. M. (2015). Mixed methods and wicked problems. Journal of Mixed Methods Research, 9(1), 3–6. https://doi.org/10.1177/1558689814562944 Munro, A., Shakeshaft, A., & Clifford, A. (2017). The development of a healing model of care for an Indigenous drug and alcohol residential rehabilitation service: A community-based participatory research approach. Health & Justice, 5(1), 1–12. https://doi.org/10.1186/s40352-017-0056-z Murray, L. K., Dorsey, S., Skavenski, S., Kasoma, M., Imasiku, M., Bolton, P., Bass, J., & Cohen, J. A. (2013). Identification, modification, and implementation of an evidence-based psychotherapy for children in a low-income country: The use of TF-CBT in Zambia. International Journal of Mental Health Systems, 7(1), 1–12. https://doi.org/ 10.1186/1752-4458-7-24 Narasimhan, S. & Chandanabhumma, P.P. (2021). A scoping review of decolonization in indigenousfocused health education and behavior research. Health Education and Behavior, the Scholars of Color Special Issue, 48(3): 306–319. https://doi. org/10.1177/10901981211010095 O’Mara-Eves, A., Brunton, G., Oliver, S., Kavanagh, J., Jamal, F., & Thomas, J. (2015). The effectiveness of community engagement in public health interventions for disadvantaged groups: A meta-analysis. BMC Public Health, 15(129). https://doi. org/10.1186/s12889-015-1352-y Pinto, R. M., Da Silva, S. B., Penido, C., & Spector, A. Y. (2012). International participatory research framework: Triangulating procedures to build health research capacity in Brazil. Health Promotion International, 27(4), 435–444. https://doi. org/10.1093/heapro/dar090 Plano Clark, V. L., & Ivankova, N. V. (2016). Mixed methods research: A guide to the field. Sage Research Methods. https://dx.doi.org/10.4135/ 9781483398341 Poth, C. N. (2020). Confronting Complex Problems with Adaptive Mixed Methods Research Practices. Caribbean Journal of Mixed Methods Research, 1(1), 29–46. https://doi.org/10.37234/CJMMR Questa, K., Das, M., King, R., Everitt, M., Rassi, C., Cartwright, C., Ferdous, T., Barua, D., Putnis, N., Snell, A. C., Huque, R., Newell, J., & Elsey, H. (2020). Community engagement intervention for communicable disease control in low- and

LEVERAGING MIXED METHODS COMMUNITY-BASED PARTICIPATORY RESEARCH

lower- middle-income countries: Evidence from a review of systemic reviews. International Journal on Equity Health, 19, 51. https://doi.org/10.1186/ s12939-020-01169-5 Salimi, Y., Shahandeh, K., Malekafzali, H., Loori, N., Kheiltash, A., Jamshidi, E., Frouzan, A. S., & Majdzadeh, R. (2012). Is community-based participatory research (CBPR) useful? A systemic review on papers in a decade. International Journal on Preventative Medicine, 3(6), 386–392. Snijder, M., Wagemakers, A., Calabria, B., Byrne, B., O’Neill, J., Bamblett, R., Munro, A., & Shakeshaft, A. (2020). ‘We walked side by side through the whole thing’: A mixed-methods study of key elements of community-based participatory research partnerships between rural Aboriginal communities and researchers. The Australian Journal of Rural Health, 28(4), 338–350. https://doi.org/ 10.1111/ajr.12655. Tervalon, M., & Murray-Garcia, J. (1998). Cultural humility versus cultural competence: A critical distinction in defining physician training outcomes in multicultural education. Journal of Health Care for the Poor and Underserved, 9(2), 117–125. https://doi.org/10.1353/hpu.2010.0233 Viswanathan, M., Ammerman, A., Eng, E., Gartlehner, G., Lohr, K. N., Griffith, D., Rhodes, S., Samuel-Hodge, C., Maty, S., Lux, L., Webb, L., Sutton, S. F., Swinson, T., Jackman, A., & Whitener, L. (2004). Community-based participatory research: Assessing the evidence (AHRQ Publication No. 04–E022–2). Agency for Healthcare Research and Quality: Evidence Report/Technology, Assessment, 99, 1–8. www.ncbi.nlm.nih.gov/ books/NBK37280/ Wallerstein, N. B., & Duran, B. (2006). Using community-based participatory research to address health disparities. Health Promotion Practice, 7(3), 312– 323. https://doi.org/10.1177/1524839906289376 Wallerstein, N., & Duran, B. (2010). Communitybased participatory research contributions to intervention research: The intersection of science and practice to improve health equity. American Journal of Public Health, 100(S1), S40–S46. https://doi. org/10.2105/AJPH.2009.184036

433

Wallerstein, N., & Duran, B. (2018). Theoretical, historical and practice roots of CBPR. In N. B. Wallerstein, B. M. Duran, J. G. Oetzel, & M. Minkler (Eds.), Community-based participatory research for health (pp. 17–30). Jossey-Bass. Wallerstein, N., Duran, B., Oetzel, J.G., & Minkler, M. (2018) On community-based participatory research. In N. B. Wallerstein, B. M. Duran, J. G. Oetzel, & M. Minkler (Eds.), Community-based participatory research for health (pp. 3–16). Jossey-Bass. Wallerstein, N., Muhammad, M., Sanchez-Youngman, S., Rodriguez Espinosa, P., Avila, M., Baker, E. A., Barnett, S., Belone, L., Golub, M., Lucero, J., Mahdi, I., Noyes, E., Nguyen, T., Roubideaux, Y., Sigo, R., & Duran, B. (2019). Power dynamics in community-based participatory research: A multiplecase study analysis of partnering contexts, histories, and practices. Health Education & Behavior, 46(supp. 1), 19S–32S. https://doi.org/10.1177/ 1090198119852998 Williams, D. R. (2012). Miles to go before we sleep: Racial inequities in health. Journal of Health and Social Behavior, 53(3), 279–295. https://doi. org/10.1177/0022146512455804 W. K. Kellogg Foundation. (2001). Community track essential program elements. www.kellogghealthscholars.org/sites/Program%20Elements.pdf. Yonas, M. A., Jones, N., Eng, E., Vines, A. I., Aronson, R., Griffith, D. M., White, B., & DuBose, M. (2006). The art and science of integrating undoing racism with CBPR: Challenges of pursuing NIH funding to investigate cancer care and racial equity. Journal of Urban Health, 83(6), 1004–1012. https://doi. org/10.1007/s11524-006-9114-x Zoellner, J. M., Porter, K. J., Brock, D. J. P., Mitchell, E. M. K., Chapman, H., Clarkston, D., Cohn, W., Hauser, L., Morris, D. W., Ramey, S. Y., Robinson, B., Schriefer, S., Voges, N., & Wiseman, K. P. (2021). Advancing engagement and capacity for rural cancer control: a mixed-methods case study of a Community-Academic Advisory Board in the Appalachia region of Southwest Virginia. Research Involvement and Engagement, 7(1), 1–16. https:// doi.org/10.1186/s40900-021-00285-y

30 Cultural Diversity in Intervention Design: A Chinese Illustrative Example Hongling Chu, Xuejun Yin and Hueiming Liu

INTRODUCTION The Unique Cultural Context of Research in China With globalization, research is increasingly conducted across multiple countries or involves stakeholders from diverse cultures. It is important to carefully consider the impact of cultural diversity in our research design, implementation and evaluation. Cultural diversity could be considered across three-levels: (a) the global research culture impacting the research question, funding and priorities; (b) the team dynamics across global research groups; and (c) the way diverse culture impacts project implementation. China is a developing country, comprising 32 provinces and cities with diverse cultures. As a developing country, China sometimes receives funding from developed countries to conduct research, which might lead to complicated team collaboration. Political support is vital for studies conducted in China. In addition, Yin–Yang philosophy and Confucianism are the predominant cultural philosophies, emphasizing the need to achieve balance (Fetters & Molina-Azorin, 2019). Considering the diverse culture and needs, achieving a balance among all parties is critical. The Yin–Yang philosophy and

Confucianism influence communication patterns and conflict management. Consistent with existing guidance for research about cultural adaptation of study results (Moore et  al., 2021), if cultural diversity is not considered, the findings may not be representative of the cultural context nor appropriately adopted to promote policy change. Further, concepts may be misunderstood. Conversely, early consideration of cultural diversity increases the potential of developing an intervention that can be widely adopted and maintained in real-world settings (Skivington et al., 2021).

Overview of the Project and Intervention Study The China Rural Health Initiative (CRHI) is a large-scale cluster randomized controlled trial conducted in 120 villages in five provinces with diverse cultures (Li et al., 2013). The overarching purpose of this trial was to identify a novel, lowcost, scalable and sustainable, community-based strategy for the prevention of blood pressurerelated diseases in rural China, which may be generalizable to a larger area if it is confirmed to be effective and feasible.

CULTURAL DIVERSITY IN INTERVENTION DESIGNS

Need and Purpose of the Chapter Complex intervention research is commonly conducted in multi-centers involving a diverse cultural environment. The theoretical framework of complex intervention research aims to help researchers work with other stakeholders to identify the key questions about complex interventions and design, and conduct research with diverse perspectives and appropriate methods. The framework divides complex intervention research into four phases: development or identification of the intervention, feasibility, evaluation and implementation. Considering cultural determinants in each phase is critical to the success of a study. However, combining the theoretical framework of complex intervention research in a diverse cultural environment using mixed methods to solve problems at each phase is still to be systematically sorted out. Therefore, this chapter aims to sort out the relevant key points to conduct an intervention mixed methods project in the different phases, and to consider and deal with the cultural diversity in each phase using qualitative or quantitative approaches.

Rationale for Using Mixed Methods Research The global research culture sets the scene for developing the research question through funding calls and mechanisms. Mixed methods research plays an integral part in this process. The World Health Organization (WHO) has set the scene by identifying the notable evidence gaps and priorities through modelling and epidemiological studies since 1990. However, as mentioned above, the global research culture influences the “evidence into implementation”. It is crucial for an organization such as the WHO to understand the local needs and culture of those for whom they are providing funding. For example, evidence generated based on studies conducted in high-income countries may not be able to be implemented in low- and middle-income countries. Therefore, it is essential to use mixed methods research to unpack the local contextual factors to co-design research with local stakeholders. This can be done through health needs assessment, Delphi studies, focus group discussions and interviews with key informants to ensure that local priorities are addressed in ways that are feasible and sustainable (Nambiar et al., 2017). Mixed methods research is particularly suitable for intervention studies that are implemented in settings with cultural diversity because it provides a practical way to understand multiple perspectives,

435

multiple stakeholders, different types of causal pathways and multiple types of outcomes. These can be boiled down to four main rationales: 1 Participant enrichment: to gain the most information from the diverse sample of participants. For an illustrative example and further discussion, see Chapter 25 (this volume). 2 Instrument validity: to ensure that the instruments used are appropriate and valuable. For a further discussion, see Chapter 20 (this volume). 3 Implementation fidelity: to assess whether the intervention or programme is being administered as intended. 4 Meaning enhancement: to maximize the interpretation of the findings, such as using qualitative measures to explain the statistical analysis or vice versa (Leech & Onwuegbuzie, 2010). For a further discussion and illustrative example, see Chapter 17 (this volume).

CULTURAL CONTEXT OF RESEARCH IN CHINA China boasts a total land area of 9.6 million square kilometers across four time zones. The total population is more than 1.4 billion and males account for 51.24 per cent. A total of 18.7 per cent of the population are adults aged 60 years old and above. The average years of education is 9.08 and the population living in rural areas accounts for 36.11 per cent. China is composed of 32 provinces with diverse cultures with their unique cuisines and dialects/languages, socioeconomic determinants such as the level of economic development, education level and differing levels of implementation of health policies such as specific chronic disease management policies (Xiong et al., 2022). These three factors might affect the intervention implementation or/ and effectiveness.

Factor 1: Distribution of the Five Provinces in China The China Rural Health Initiative project was conducted in five of China’s Northern Provinces utilizing collaborations established with local academic institutions and governments. The five provinces included are Liaoning, Hebei, Shanxi, Shaanxi and Ningxia (Figure 30.1). These provinces have high

436

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

rates of vascular disease with a substantial burden caused by high blood pressure attributed to an average 24-hour urinary sodium excretion of 260 mmol (equals to about 15 g of salt). However, the diet and culture are different among these five provinces (see Table 30.1).

Factor 2: Developing a Research Team with Cultural Diversity The research team members consist of various backgrounds, including experts who majored in epidemiology, health education, qualitative research,

Figure 30.1  Distribution of the five provinces in China Table 30.1  The diet and culture in five provinces in China Province

Population

Location in China

Diet structure

Taste

Liaoning

42.59 million

Northeast

Fat, fishy and salty

Hebei

74.61 million

North

Shanxi

34.92 million

North

Shaanxi

39.52 million

Center

Ningxia

7.20 million

Northwest

Staple foods are mostly rice, wheat, millet, corn and sorghum Staple foods are mostly wheat food made of flour, such as the various kinds of noodles type of high-grain diet mainly based on the carbon source and heat energy of food Beef and mutton are the main food

Salty and heavy oil Pickled vegetables

Salty

Islamic diet

CULTURAL DIVERSITY IN INTERVENTION DESIGNS

statistics and management, and they are from international institutions as well as local provincial organizations (see Figure 30.2). Team membership is an important aspect of mixed methods research and can be determined by the dominance of different components within a study. Methodological respect between team members and a principal investigator who values integration emerged as essential to achieving integrated research outcomes (O’Cathain et al., 2008). High-quality mixed methods intervention projects require good collaborations between stakeholders who are involved in developing and implementing intervention. These can be difficult as stakeholders may have professional or personal interests in portraying the intervention positively or see evaluation as threatening.

437

interventions in different environments is of key concern, especially in resource-limited settings. Team training is the critical challenge in such a large-scale project involving implementers from the levels of province, county, township and village with different education backgrounds and work experiences. It is unrealistic to take implementers from all the levels to be trained in Beijing. In the phase of evaluation, quantitative data alone is not enough to understand why the intervention works, what are the contexts, barriers and facilitators impacting implementation, and how effective is intervention implemented in complex settings.

THE ILLUSTRATIVE PROJECT Factor 3: Challenges Encountered In the phase of development, how to develop an intervention that is simple, understandable, feasible and suitable for different provinces with cultural diversity is worth considering. The work and rest habits of rural areas and local customs and culture varied in different regions. Therefore, the intervention should generally be the same, but adapted for various places. In the phase of implementation, ensuring a high-quality implementation of complex

Objectives The overarching purpose of this cluster-randomized controlled trial aimed to identify a novel, low-cost, scalable and sustainable, community-based strategy for preventing blood pressure-related diseases in rural China (Li et  al., 2013). We used mixed methods to evaluate the fidelity, delivery, reach, and contextual factors of intervention, and to explore the feasibility of the scale of the complex intervention in other settings.

Figure 30.2  The team consists of global and local research members

438

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Theoretical framework

Design

Process evaluation is designed to provide an indepth understanding of the study context and results to inform policy and practice. As indicated in Figure 30.3, this is achieved by examining the impact and appropriate contextual application of the intervention. Context includes anything external to the intervention that may bar or promote its facilitation or effects. Blue boxes depict the key components of a process evaluation. Investigation of these components is shaped by a clear intervention description and informs interpretation of outcomes (Figure 30.3). As described above, implementation will often vary from one context to another. However, an intervention may have different effects in different contexts even if its implementation does not vary. Complex interventions work by introducing mechanisms that are sufficiently suited to their context to produce change, while causes of problems targeted by interventions may differ from one context to another. Understanding context is therefore critical in interpreting the findings of a specific evaluation and generalizing beyond it. Even when an intervention itself is relatively simple, its interaction within its context may still be highly complex.

We used a scaffolded mixed methods design (Fetters, 2020) which is an integrated mixed methods intervention approach incorporating the framework of developing and evaluating complex interventions. The design can be used to develop interventions suited to the various study settings across different cultures, evaluate the effectiveness of an intervention, and also explain the mechanisms by process evaluation. Figure 30.4 outlines how we integrate the mixed methods interventional study design and key elements of the development and evaluation process. We proposed an implementation matrix based on scaffolded mixed methods (see Figure 30.5).

Phase 1 Development 1 To explore the context/culture of the target population. Site information sheets and feasibility questionnaires with several qualitative questions were distributed to several potential sites to explore the context/culture of target sites and population at the early stage of this project.

Figure 30.3  Key functions of process evaluation and relations among them Source: Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., … & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. BMJ, 350. https://doi.org/10.1136/bmj.h1258

CULTURAL DIVERSITY IN INTERVENTION DESIGNS

439

Figure 30.4  Scaffolded mixed methods design integrated mixed methods interventional approach with a framework of developing and evaluating complex interventions

Figure 30.5  Implementation matrix focused on culture diversity based on scaffolded mixed methods 2 To develop interventions based on a theory. In this diverse research team, there are health education professionals who are proficient at applying theoretical guidance when formulating health education and health promotion interventions. In this study, the health belief model (Rosenstock et  al., 1988) was used to develop interventions as shown in Table 30.2. Among these components of the intervention, all the health education materials were specially designed for this project. They are simple and easy to understand for people living in rural

areas, due to the education around sodium intake being spread through local, traditional art forms. For example, the knowledge or skills about sodium reduction was incorporated into local theatres or songs. 3 To conduct a feasibility/pilot study. Both health education materials and manual operations for organizing health education activities were piloted in rural areas with comparable socioeconomic status. Qualitative data was collected and analyzed to provide information on acceptability and suggestions for optimizing intervention

440

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 30.2  The health belief model (HBM) and derived intervention used in the example project Domains of HBM

Features of the intervention in this project Intervention according to the HBM

1. Perceived susceptibility, severity

Improving the awareness of: a.  How much salt is consumed b.  High salt intake worsens health and diseases caused by high salt intake Providing the information and cues about: a.  The benefits of consuming less salt b.  The effects of low sodium salt c.  Removing concerns that low salt intake would result in no energy to work d.  To create a salt-reduction atmosphere and tell them how to reduce salt intake (knowledge and behaviors) Organizing activities to help them improve their self efficacy: a.  To carry out activities to encourage their beliefs and behaviors for salt reduction

2. Perceived benefit, barriers, cues to action

3. Perceived self-efficacy

a.  Health education materials - Posters - Calendars - Stickers for placing on salt containers b.  Health education activities - Program launch events - Activities organized in consideration of local context - Activities for individuals at elevated risk of cardiovascular disease - Student-to-parent education activities c.  Low sodium salt substitute supply

Source: Chu, H., Zhang, J., Fetters, M. D., Niu, W., Li, H., Li, N., … & Wu, Y. (2021). A Mixed Methods Process Evaluation of a Clustered-Randomized Controlled Trial to Determine the Effects of Community-Based Dietary Sodium Reduction in Rural China. Frontiers in Medicine, 8, 646576.https://doi.org/10.3389/fmed.2021.646576

components. This was an iterative process to pilot and optimize interventions until appropriate, adapt to the context of target sites, and consider generalization for targeted sites in northern China with varying dietary cultures.

Phase 2 Implementation In such a complex and large-scale study setting, you may encounter many unexpected situations during implementation. Therefore, it is critical to train, communicate with and monitor the staff who are implementing the intervention. First, conducting efficient training is essential. The train-the-trainer model provides organizations with an efficient way to roll out training at scale (Anderson & Taira, 2018). The train-the-trainer model works by having an expert train internal employees to become trainers in a specific subject. These internal employees, now trainers, can train others within the organization, using what they have learned. In our study, facilitators at county levels were trained by project experts, and then they trained the implementers at township or village levels. In addition, communication among team members at all levels is very important. Timely communication helps solve problems in the process of

project implementation in a timely manner. In our study, two full-time research assistants are responsible for the coordination and communication during the research process, and each county has a research coordinator to ensure the progress of the project. Third, it is important to monitor the clinical trial’s progression by ensuring that it is conducted, recorded and reported in accordance with the protocol’s standard operating procedures (SOPs), good clinical practice (GCP), and applicable regulatory requirements. Ongoing qualitative and quantitative data collection and analysis across the project’s progression is necessary to ascertain research appropriateness.

Phase 3 Evaluation The objective and quantitative evaluation for the effectiveness of complex interventions is the primary aim, which is critical to provide evidence for policy-making and decision-making. In our study, the primary outcome was the mean daily sodium consumption calculated from 24-hour urine. Besides the quantitative outcome evaluation, increasing attention is given to process evaluation, especially for a large-scale trial influenced by

CULTURAL DIVERSITY IN INTERVENTION DESIGNS

various cultures. Process evaluation can be carried out during the implementation process to find and solve problems in time. It can also be carried out after the research to interpret the quantitative findings, as well as provide a reference for the scaleup in other settings. In our study, the mixed methods process evaluation was used to investigate the implementation and to evaluate feasibility of the complex intervention to translate the findings from a clinical study to the real world. A convergent mixed methods process evaluation design was used in this study (Chu et al., 2021). Quantitative data were collected from activity logs and routine study records. Qualitative data were collected from 53 project stakeholders and 45 villagers from 10 intervention villages. Thematic analysis of qualitative interviews facilitated integration with the descriptive quantitative data analysis based on theory-informed domains of fidelity, delivery, reach, receipt and contextual factors of intervention from a process evaluation framework.

Results: Qualitative, Quantitative, Integrated Qualitative. In the phase of development, qualitative results from a pilot study to obtain feedback from participants and implementers were used to modify the health education materials and activities. Unfortunately, we did not have a formal collection and analysis of the data from individual interviews and group discussions. In the phase of evaluation, qualitative results were mainly reflected in the evaluation of the receptibility of the health education materials and activities, as well as context. Contextual factors hindering full uptake of the intervention included a preference for a salty taste, the higher cost of low-sodium salt and low education levels of the villagers. This process evaluation indicated that conducting health education interventions in rural areas requires policy and administrative support, enthusiastic staff, easy-to-understand health education materials and activities, stakeholders and tempered expectations as behavioral change requires time. Quantitative. For the main study, after complex interventions were conducted for 18 months, 1,903 people had valid 24-hour urine collections. The mean urinary sodium excretion in the intervention group compared to the villages that served as the controlled group was reduced by 5.5 per cent. In the intervention group, potassium excretion increased by 16 per cent and the sodium-topotassium ratio declined by 15 per cent. Between the intervention and control groups, the mean

441

blood pressure differences were –1.1 mm Hg systolic and –0.7 mm Hg diastolic. The difference in proportion with hypertension was –1.3 per cent. The absence of the effects on blood pressure reflected the moderate changes in sodium and potassium intake achieved (Li et al., 2016). For process evaluation, quantitative results mainly described fidelity, delivery and reach for each health intervention. A total of 5,450 sheets of posters, 31,400 calendars and 78,000 sheets of stickers were delivered as planned, and 11 promotion activities were conducted in each village (Chu et al., 2021). Integrated. Integration occurred in the phase of design for the whole interventional project. For the process evaluation, integration occurred in design, data collection, data analysis and joint display based on the framework of process evaluation. The mixed data were analyzed based on the study process evaluation framework using the process of joint display analysis (Fetters, 2020; Guetterman et  al., 2015). This iterative, juxtaposing process links quantitative and qualitative data, examines the resulting implications, and draws conclusions in the light of both types of findings (see Table 30.3). Overall, there were clear differences in population sodium and potassium intake between villages that were most likely a consequence of increased use of salt substitutes. The absence of the effects on blood pressure reflects the moderate changes in sodium and potassium intake achieved. In conclusion, this multifaceted intervention was implemented well and effectively in rural China. The process evaluation has indicated that conducting health education interventions in rural areas requires policy and administrative support, enthusiastic staff, easy-to-understand health education materials and activities, stakeholders and tempered expectations as behavioral change requires time.

IMPLICATIONS The illustrative project demonstrates the feasibility and benefits of using mixed methods process evaluation in an implementation study involving cultural diversity. We highlighted how the scaffold design is appropriate in integrating mixed methods interventional evaluation with the framework for complex intervention development and evaluation. Moreover, it is undeniable that, in a broad global context, larger cultural and language differences need to be considered. Therefore, keen attention to culture at each phase of research is critical, especially in the following five aspects.

High 5,450/5,450 units

Fidelity (to protocol) Delivery (planned/ delivered) Reach

Recommended and accepted by villagers.

Meta inferences

Stickers for placing on salt containers

Recommended

Deemed practical

43/45 interviewees

High 60/60 events

Program launch events

Recommended

Strongly recommended

Served to remind Created a specific when using salt and suitable atmosphere for salt reduction

High High 31,400/31,400 units 78,000/78,000 units 40/45 interviewees 39/45 interviewees

Calendars

Moderate 60/120 events

1,595 children’s worksheets Some villagers felt Some elders or people Primary school the content with CVD couldn’t students were became participate interested in repetitive and these activities wanted more variety of information Strongly Moderately Strongly recommended recommended recommended

High 240/228 events

22/32 interviewees 12/13 interviewees

High 300/300 events

High NA/115,228 bags* 37/45 interviewees Some felt the taste was less salty than regular salt, and it was more expensive Recommended

Activities organized Activities for individuals Student-to-parent Low sodium salt in consideration of at elevated risk of education activities substitute supply local context cardiovascular disease

Source: Chu, H., Zhang, J., Fetters, M. D., Niu, W., Li, H., Li, N., … & Wu, Y. (2021). A Mixed Methods Process Evaluation of a Clustered-Randomized Controlled Trial to Determine the Effects of Community-Based Dietary Sodium Reduction in Rural China. Frontiers in medicine, 8, 646576.https://doi.org/10.3389/fmed.2021.646576

Being popular, easy-tounderstand and simple

Receipt

40/45 interviewees

Posters

Dimension

Table 30.3  Joint display of fidelity, delivered, reach, receipt and meta inferences by each component of intervention.

442 THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

CULTURAL DIVERSITY IN INTERVENTION DESIGNS

443

Intervention Development: More Fully Understand Stakeholders’ Feedback

Implementation: Incorporating Process Evaluation to Tailor Implementation Strategies

In the intervention development phase, it was necessary to fully consider economic level, educational background and other local environments, as well as the feasibility of interventions. Closed and open questionnaires were used to learn more about the above situation. According to the survey results, we found that the villagers would mind the minor price difference between low-sodium and regular salt, even though only RMB ¥1 (equal to USD$0.15). Therefore, we designed an element of intervention to provide a subsidy for purchasing low-sodium salt. Other interventions also took the pilot study to obtain the stakeholders’ feedback to make recommended modifications. A feasibility study should be designed to assess predefined progression criteria that relate to the evaluation design (e.g., reducing uncertainty around recruitment, data collection, retention, outcomes and analysis) or the intervention itself (e.g., optimal content and delivery, acceptability, adherence, the likelihood of cost-effectiveness or the capacity of providers to deliver the intervention). If the programme theory suggests that contextual or implementation factors might influence the acceptability, effectiveness or cost-effectiveness of the intervention, these questions should be considered. Depending on the results of the feasibility study, further work might be required to progressively refine the intervention before embarking on a full-scale evaluation (Skivington et al., 2021).

In the illustrative project, thematic analysis of qualitative interviews facilitated integration with the descriptive quantitative data analysis based on theory-informed domains of fidelity, delivery, reach, receipt and contextual factors of intervention from a process evaluation framework. This process evaluation has indicated that conducting health education in rural areas requires policy and administrative support, enthusiastic staff, easy-tounderstand health education materials and activities, stakeholders and tempered expectations as behavior change requires time. Process evaluations aim to provide a more detailed understanding needed to inform policy and practice through examining aspects such as implementation, mechanisms of impact and context. The effects of a complex intervention might often be highly dependent on cultural factors, such that an intervention that is effective in some settings could be ineffective or even harmful elsewhere (Craig et  al., 2018). Context can be considered as both dynamic and multidimensional. Key dimensions include physical, spatial, organizational, social, cultural, political or economic features of the healthcare, health system or public health contexts in which interventions are implemented.

Intervention Fidelity and Cultural Adaptation Fidelity could be evaluated by qualitative or/and quantitative data and results. Positive outcomes can sometimes be achieved even when an intervention was not delivered fully as intended. Hence, to begin forming conclusions about what works, process evaluation will usually aim to capture fidelity (whether the intervention was delivered as intended) and dose (the quantity of intervention implemented). Moreover, complex interventions usually undergo some tailoring when implemented in different contexts. Capturing what is delivered in practice, with close reference to the theory of the intervention, can enable evaluators to distinguish between adaptations to make the intervention fit different contexts and changes that undermine intervention fidelity (Moore et al., 2015). For an additional example, see Chapter 27 (this volume).

Implementation: Ensuring Strong Reporting when Using a Train-theTrainer Approach In our illustrative project, implementers at county levels were trained by project experts, then they trained the implementers at township or village levels. Such a model could adapt the training content at both township and village levels. Meanwhile, the effectiveness should be evaluated using quantitative exams and qualitative interviews to ensure the quality of training. The trainthe-trainer model is a promising paradigm for the propagation of resuscitation training in limited resource settings. A systematic review found clear evidence that train-the-trainer programming improves the knowledge and skills of providers. In addition, it found that intimate knowledge of the setting (language, culture, economic and resource barriers) and support from the ministry of health of the country are two overarching factors that facilitate successful train-the-trainer programming in limited resource settings (Anderson & Taira, 2018).

444

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Culturally Sensitive Implementation: Considering Folk Activities is Integral Considering folk activities is one element of the intervention in this illustrative example. In the phase of development, closed and open questionnaires were used to learn about the feasibility for each potential intervention. According to survey results, one kind of health education activity was designed to be organized according to local culture. That means the implementers could organize these kinds of activities combined with some other traditional cultural activities, such as embedding knowledge or skills about sodium reduction into traditional drama, songs or Chinese traditional music playing. This could improve the feasibility for these complex interventions among such large-scale areas with cultural diversity.

CONCLUSION To conclude, we advance that the researcher has to incorporate cultural diversity by considering the context of global cultural background, and the construction of a multicultural research team to lay a solid foundation for research across diverse cultures. When conducting complex interventions in a diverse cultural environment, it is critical to explore the research setting at the design phase to develop appropriate intervention and implementation procedures within that complex environment. Because such a multicultural project is usually large scale, it is necessary to integrate qualitative and quantitative research to understand the basic situation of various places; conduct a pilot study for intervention and implementation procedures; and to provide effective optimization suggestions. In addition, it is necessary to conduct an objective and quantitative evaluation of the primary outcomes, as well as to evaluate the barriers and facilitators of interventions in this diversified cultural background for future use. Cultural diversity is inevitable within most multicenter intervention studies. Therefore, it is far from enough to only collect data of quantitative outcomes to evaluate the effectiveness. Moreover, it is vital to identify the feasibility of the intervention suited to diverse cultures and to take account of how it interacts within culturally diverse settings. Thus, these questions can only be solved by integrating qualitative and quantitative approaches. The integration of qualitative and quantitative methods might solve a focused

question using mixed methods during the phase of development, implementation or evaluation— e.g., by administering a standard survey questionnaire and then asking for in-depth explanations, or by using qualitative measures to explain the statistical analysis or vice versa. We might be standing at the forefront of designing a mixed methods project that can make seemingly complex problems become logical and generate compelling evidence for the process and results through the use of mixed methods research.

WHAT TO READ NEXT Liu, Y., Chu, H., Peng, K., Yin, X., Huang, L., Wu, Y., … & Liu, H. (2021). Factors associated with the use of a salt substitute in rural China. JAMA Network Open, 4(12), e2137745-e2137745. doi:10.1001/ jamanetworkopen.2021.37745

A careful assessment of contextual factors and human behaviour is essential when implementing population health strategies. The study described in this article aimed to understand the contextual factors and human behaviours associated with the use of salt substitutes and to provide insight into the variation in the trial’s interim results, with a view to identifying the potential barriers to and facilitators of the large-scale population use of salt substitutes outside of the trial setting. Moore, G., Campbell, M., Copeland, L., Craig, P., Movsisyan, A., Hoddinott, P., … & Evans, R. (2021). Adapting interventions to new contexts— the ADAPT guidance. BMJ, 374. p. n1679. https:// doi.org/10.1136/bmj.n1679

Although some interventions transfer well, effectiveness and implementation often depend on the context. Achieving a good fit between intervention and context then requires careful and systematic adaptation. This paper presents new evidence and consensus informed guidance for adapting and transferring interventions to new contexts. Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., … & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. BMJ, 350. https://doi. org/10.1136/bmj.h1258

This guidance recognized the value of process evaluation within trials, stating that it “can be used to assess fidelity and quality of implementation, clarify causal mechanisms and identify contextual factors associated with variation in outcomes” (p. 1).

CULTURAL DIVERSITY IN INTERVENTION DESIGNS

REFERENCES Allen, L. N., Pullar, J., Wickramasinghe, K. K., et al., (2018). Evaluation of research on interventions aligned to WHO ‘Best Buys’ for NCDs in lowincome and lower-middle-income countries: a systematic review from 1990 to 2015. BMJ Global Health, 3(1), e535. http://doi.org/10.1136/bmjgh2017-000535 Anderson, C. R., & Taira, B. R. (2018). The train the trainer model for the propagation of resuscitation knowledge in limited resource settings: A systematic review. RESUSCITATION, 127, 1–7. http://doi. org/10.1016/j.resuscitation.2018.03.009 Chu, H., Zhang, J., Fetters, M. D., et  al. (2021). A mixed methods process evaluation of a clusteredrandomized controlled trial to determine the effects of community-based dietary sodium reduction in rural China. Frontiers in Medicine, 8. 646576. https://doi.org/10.3389/fmed.2021.646576 Craig, P., Di Ruggiero, E., Frohlich, K. L., Mykhalovskiy, E., & White, M. (2018). Taking account of context in population health intervention research: guidance for producers, users and funders of research. NIHR Journals Library. https://doi.org/ doi:10.3310/CIHR-NIHR-01 Fetters, M. D. (2020). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. SAGE. Fetters, M. D., & Molina-Azorin, J. F. (2019). A call for expanding philosophical perspectives to create a more “worldly” field of mixed methods: The example of Yinyang philosophy. Journal of Mixed Methods Research, 13(1), 15–18. https://doi.org/ 10.1177%2F1558689818816886 Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays, Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm.1865 Leech, N. L., & Onwuegbuzie, A. J. (2010). Guidelines for conducting and reporting mixed research in the field of counseling and beyond. Journal of Counseling and Development, 88(1), 61–69. https://doi.org/10.1002/j.1556-6678.2010. tb00151.x Li, N., Yan, L. L., Niu, W., et al., (2013). A large-scale cluster randomized trial to determine the effects of community-based dietary sodium reduction–the China Rural Health Initiative Sodium Reduction Study, American Heart Journal, 166(5), 815–822. http://doi.org/10.1016/j.ahj.2013.07.009

445

Li, N., Yan, L. L., Niu, W., et al., (2016). The effects of a community-based sodium reduction program in rural China – A cluster-randomized trial. PLoS One, 11(12), e166620. http://doi.org/10.1371/journal. pone.0166620 Liu, H., Huffman, M. D., & Trieu, K. (2020). The role of contextualisation in enhancing non-communicable disease programmes and policy implementation to achieve health for all. Health Research Policy and Systems, 18(1), 38. http://doi. org/10.1186/s12961-020-00553-5 Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonnell, C., Hardeman, W., Moore, L., O’Cathain, A., Tinati, T., Wight, D., & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. BMJ, 350. https://doi. org/10.1136/bmj.h1258 Moore, G., Campbell, M., Copeland, L., Craig, P., Movsisyan, A., Hoddinott, P., Littlecott, H., O’Cathain, A., Pfadenhauer, L., Rehfuess, E., Segrott, J., Hawe, P., Kee, F., Couturiaux, D., Hallingberg, B., & Evans, R.(2021). Adapting interventions to new contexts-the ADAPT guidance. BMJ, 374, n1679. http://doi.org/10.1136 /bmj.n1679 Nambiar, B., Hargreaves, D. S., Morroni, C., Heys, M., Crowe, S., et  al. (2017). Improving health-care quality in resource-poor settings. Bulletin of the World Health Organization, 95(1), 76–78. http:// doi.org/10.2471/BLT.16.170803 O’Cathain, A., Murphy, E., & Nicholl, J. (2008). Multidisciplinary, interdisciplinary, or dysfunctional? Team working in mixed-methods research. Qualitative Health Research, 18(11), 1574–1585. http:// doi.org/10.1177/1049732308325535 Rosenstock, I. M., Strecher, V. J., & Becker, M. H. (1988). Social learning theory and the Health Belief Model. Health Education Quarterly, 15(2), 175–183. http:// doi.org/10.1177/109019818801500203 Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., Petticrew, M., Roycroft-Malone, J., White, M., & Moore, L. (2021). A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ, 374, n2061. http://doi. org/10.1136/bmj.n2061 Xiong, S., Cai, C., Jiang, W., et  al., (2022). Primary health care system responses to non-communicable disease prevention and control: A scoping review of national policies in Mainland China since the 2009 health reform. The Lancet Regional Health. Western Pacific, 100390. http://doi. org/10.1016/j.lanwpc.2022.100390

31 Examining the Influences of Spanish Research Culture in Systematic Observation with Mixed Methods M . Te r e s a A n g u e r a , E u l à l i a A r i a s - P u j o l , Francisco Molinero and Luca Del Giacco

INTRODUCTION This work aims to provide the reader with an insight into the Spanish research culture and its influence on observation as mixed methods and also facilitate the realization of observational studies in the framework of the mixed methods. The research culture in the Spanish context has been characterized by the application of systematic observation through the use of observational methodology, fostering the development of mixed methods in our context through a peculiar integration of the qualitative and quantitative moments, and from two complementary perspectives. On one hand, we started from a broad conceptual and methodological perspective to consider the apparent variability existing in approaches, conceptual frameworks, individual differences between participants and the plurality of multiple elements that characterize the profile of each case. On the other hand, we considered the multitude of possibilities allowed by mixed method studies and the combination of flexibility and rigour that characterizes systematic observation. We have included two examples (psychotherapy and mediation) in which this methodology has been gaining ground in recent years, highlighting its potential application in improving the effectiveness of professional interventions.

The Spanish cultural and research context The research context to which we refer focuses on the observational methodology, which in the last four decades has been developed incessantly in Spain, and which has opened up a wide range of possibilities for the treatment of systematic observation, especially at the research and professional levels, but with repercussions in the cultural context. From a research perspective, Hanover Research (2012) states that a research culture has formed as a supportive context in which research is expected, discussed, produced and evaluated uniformly. In essence, it is a system of widely shared and strongly sustained values that requires that the activity carried out be evaluated by external and neutral people and institutions. This period began in Spain in November 1975 before any precedents in Spain or other European countries existed. The beginnings were very hard, and we were gradually gathering information from the USA, which was always our inspiration. Throughout the first 20 years of very intense work, we had PhD students interested in observational methodology, and from various Spanish universities. This led us to co-direct many doctoral theses that had in common the use of the observational

EXAMINING THE INFLUENCES OF SPANISH RESEARCH CULTURE

methodology at the same time that the number of publications and presentations in congresses increased and the research group was consolidating. Likewise, just beginning the century and millennium, our relationship grew with colleagues from Iceland, Portugal, Italy and Germany, as well as from some Latin American countries (Mexico, Colombia, Chile, Brazil). The efforts of the first period in the Spanish context focused on the definition of a taxonomy (Anguera, 1979) and the construction ex novo of procedural systematics oriented to the creation of an observational methodology as a scientific method.

Systematic Observation: The Methodology Adaptable to an MMR Design in Spanish Research Culture The profile of the observational methodology, which allows scientifically conducting systematic observation studies, and from the mixed methods approach, was specified in the study of spontaneous perceptible behaviour and in habitual contexts (Anguera, 2003). Direct and indirect observation were differentiated (Anguera et al., 2018), taking into account that direct observation is based on visual perception, while indirect observation is based on information issued verbally, vocally or from various documents (self-reports, in-depth interviews, graphs, photographs, diaries, etc.), and both were developed. Eight observational designs were proposed to guide and organize the investigation, as a result of the intersection between three dichotomous criteria: idiographic–nomothetic, punctual–follow-up and unidimensional–multidimensional (Blanco-Villaseñor et  al., 2003). Each study (or each specific objective of a study) is assigned the appropriate observational design. The sequencing of stages was deployed from an initial design approach. Once the design has been assigned, the custom construction (ad hoc) of an observational instrument must be addressed, which must be adapted to the theoretical framework, context and situation of interest to study (Anguera et al., 2007). There are three basic types: category system, field format and combination of field format and category systems. The observational instrument most commonly used is the combination of field format and category systems (Sánchez-Algarra & Anguera, 2013). To do this, the dimensions must be identified, which constitute the respective facets of the problem we are studying and which interact with each other in the episodes of reality. It is highly recommended to raise the dimensions in such a way that they are consistent with the theoretical

447

framework in which we locate ourselves. For each of the dimensions, a mutually exclusive list of behaviors will be prepared, with its corresponding code. Therefore, each behaviour unit will have as many codes, at its maximum, as dimensions have been proposed. It may happen that a dimension is displayed in subdimensions of first level, second level, etc. In this case, it will be coded according to the most molecularized level of subdimensions (Anguera, 2021). To obtain the coded record, the criteria for segmenting the flow of behavior into units must initially be considered, and options have been proposed to facilitate it for both direct (Anguera & Izquierdo, 2006) and indirect observation (Anguera, 2021). As a consequence, the observational record will have a structure in the form of a code matrix, where each row will contain the codes corresponding to the co-occurring behaviours that correspond to the different dimensions/subdimensions, and the succession of rows is determined by the order of behavioural units. This order is of crucial importance to materialize the quantitizing that is at the centre of the QUAL-QUAN-QUAL approach that allows the observational methodology to be considered as a mixed method in itself. Work has been done to ensure that an adequate coding is available in observational studies, adapting to the various possibilities of observational sampling (instantaneous or interval) if a continuous record cannot be obtained. This stage would be followed by data quality control and data analysis. The guarantee of objectivity had its own development, exploring the minimization of biases. The most relevant being the reactivity bias, when the observed person detects that they are being recorded or observed, and the expectation bias, on those occasions when the observer does not adequately contextualize what they perceive or even unconsciously alters their interpretation of what is perceived by the attractive force of the theoretical framework to which they are subject. There are very diverse possibilities in data quality control (Blanco-Villaseñor et  al., 2010), both from a deployment of concordance and agreement, and of some types of validity, and the application and adaptation of the generalizability theory (Blanco-Villaseñor et al., 2014). This data quality control is a very relevant step for acquiring the mark of scientificity. In this same direction, work was done on observer training (Losada & Manolov, 2015) and on methodological quality (Chacón-Moscoso et al., 2019; Portell et al., 2015). Since the end of the last century, computer programs were developed (Table 31.1) that aimed to definitively banish manual records, and it constitutes an important milestone that has been optimized over recent years, all being free.

Authors

Hernández-Mendo, A., Ramos, R., Peralbo, M., & Risso, A. (1993). Un programa para el análisis observacional: Transcriptor v1.1, aplicación en psicología del deporte [A program for observational analysis: Transcriber v1.1, application in sports psychology]. Revista de Entrenamiento Deportivo, 7, 18–25.

Hernández-Mendo, A., Anguera, M. T., & Bermúdez-Rivera, M. A. (2000). Software for recording observational files. Methods, Instruments & Computers, 32(3), 436–445. https://doi. org/10.3758/bf03200813

Castellano, J., Perea, A., & Alday, L. (2005). Match Vision Studio. Software para la observación deportiva [Sports observation software]. In L. M. Sautu, J. Castellano, A. Blanco-Villaseñor, A. Hernández-Mendo, A. Goñi & F. Martínez (Eds.), Evaluación e intervención en el ámbito deportivo. Diputación Foral de Álava.

Castellano, J., Perea, A., Alday, L., & Hernández-Mendo, A. (2008). The Measuring and Observation Tool in Sports. Behavior Research Methods, 40(3), 898–905. https://doi.org/10.3758/ brm.40.3.898

Bakeman, R., & Quera, V. (1996). Análisis de la interacción. Análisis secuencial con SDIS y GSEQ [Analysis of the interaction. Sequential analysis with SDIS and GSEQ]. Ra-Ma.

Bakeman, R., & Quera, V. (2011). Sequential analysis and observational methods for the behavioral sciences. Cambridge University Press.

Jonsson, G. K., Anguera, M. T., Blanco-Villaseñor, A., Losada, J. L., Hernández-Mendo, A., Arda, T., Camerino, O., & Castellano, J. (2006). Hidden patterns of play interaction in soccer using SOF-CODER. Behavior Research Methods, Instruments and Computers, 38(3), 372–381. https://doi.org/10.3758/bf03192790

Hernández-Mendo, A., López-López, J. A., Castellano, J., Morales-Sánchez, V., & Pastrana, J. L. (2012). Hoisan 1.2: programa informático para uso en metodología observacional [Hoisan 1.2: software for observational methodology]. Cuadernos de Psicología del Deporte, 12(1), 55–78. https://doi.org/10.4321/S1578-84232012000100006

Software

TRANSCRIPTOR

CODEX

MATCH VISION STUDIO

MOTS

SDIS-GSEQ

GSEQ5

THEMECODER

HOISAN

Table 31.1  Software designed from the framework of the Spanish research culture

Registro, control de calidad del dato, análisis secuencial de retardos, y análisis de coordenadas polares

Registro

Registro, control de calidad del dato, y análisis secuencial de retardos

Registro, control de calidad del dato, y análisis secuencial de retardos

Registro

Registro

Registro

Registro

Aim

www.menpas.com

http://bakeman.gsucreate. org/

www.menpas.com

Download

448 THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Soto-Fernández, A., Camerino, O., Iglesias, X., Anguera, M.T., & Castañer, M. (in press). LINCE PLUS Software for Systematic Observation Studies of Sports and Health. Behavior Research Methods. Published online September 21 2021. https://doi.org/10.3758/s13428-021-01642-1

Barreira, D., Garganta, J., Castellano, J., & Anguera, M. T. (2013). SoccerEye: A Software solution to observe and record behaviours in sport settings. The Open Sports Sciences Journal, 6(1), 47–55. https://doi.org/10.2174/1875399x01306010047

Hernández-Mendo, A., Blanco-Villaseñor, A., Pastrana, J. L., Morales-Sánchez, V., & RamosPérez, F.J. (2016). SAGT: Aplicación informática para análisis de generalizabilidad [Computer application for generalizability analysis]. Revista Iberoamericana de Psicología del Ejercicio y el Deporte, 11(1), 77–89.

Rodríguez-Medina, J., Arias, V., Arias, B., Hernández-Mendo, A., & Anguera, M. T. (2019). Polar Coordinate Analysis, from HOISAN to R: A Tutorial Paper [Unpublished manuscript] https:// jairodmed.shinyapps.io/HOISAN_to_R/

LINCE PLUS

SOCCER EYE

SAGT

R modules

Source: Author created.

Gabin, B., Camerino, O., Anguera, M. T., & Castañer, M. (2012). Lince: Multiplatform sport analysis software. Procedia - Social and Behavioral Sciences, 46, 4692–4694. https://doi.org/10.1016/j. sbspro.2012.06.320

LINCE

Optimización graficaciòn de vectores en análisis de coordenadas polares

Análisis de generalizablidad

Registro

Registro y control de calidad del dato

Registro y control de calidad del dato

https://jairodmed. shinyapps.io/HOISAN_ to_R_2021/

www.menpas.com

https://observesport.github. io/lince-plus/

http://lom.observesport. com

EXAMINING THE INFLUENCES OF SPANISH RESEARCH CULTURE 449

450

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Having computer support has allowed us to cover a new target in our research culture, allowing a wide deployment and application of quantitative data analysis techniques. Among the various analytical techniques that can be used from qualitative data (Table 31.2)—an issue that we consider very important in the mixed methods approach—three stand out: lag sequential analysis, polar coordinate analysis and T-pattern detection. For further discussion of how technology can support analysis, see also Chapter 22 (this volume). These three quantitative analysis techniques based on qualitative data constitute a Gordian knot in the majority of studies of research culture (Anguera et  al., 2021), without forgetting other multivariate analyses that are carried out, such as log-linear, logistic regression, decision tree, etc. In the last stage, in which we find ourselves, the culture of research has shown a special interest in mixed methods, both in conceptual aspects (Anguera et  al., 2018; Johnson et  al., 2007) and in its fit into the observational methodology (Izquierdo & Anguera, 2021).

Rationale for using MMR and its Design As we pointed out in Anguera et al. (2017), in the early years of the history of mixed methods, systematic observation played an anecdotal role; however, we made a proposal for transformation from descriptive data to quantifiable code matrices that allowed a true integration of qualitative and quantitative elements. Our approach focuses on an innovative methodological proposal that allows progress on the path of mixed methods integration (O’Cathain, 2010): we aim to show how connecting (Creswell & Plano Clark, 2017) allows for the substantiation and updating of quantitizing (Sandelowski et al., 2009). All this consolidates previous proposals that have allowed us to verify how observational methodology (direct and indirect) can be considered as a mixed method in itself (Anguera et al., 2017; Anguera & Hernández-Mendo, 2016). According to our methodological proposal, this is done through an approach in which neither methods nor techniques nor data from different sources are combined, but the information is transformed in a different way from that described in the classical mixed methods literature. This proposal is based on a phrase by Creswell & Plano Clark (2017) that we especially value: There are three ways in which mixing occurs: merging or converging the two datasets by actually bringing them together, connecting the two

datasets by having one build on the other, or embedding one dataset within the other so that one type of data provides a supportive role for the other dataset. (p. 7; the underlining is ours)

For a further discussion about their evolutions in design thinking and approaches, see Chapter 2 (this volume). This mixing, in the connecting option, taken both literally and from a broader perspective, constitutes a central fulcrum for a rethinking of the quantitizing. Indeed, literally speaking, “connecting the two datasets by having one build on the other” will imply that one database—which is qualitative in nature—can give rise to another through its transformation. This transformation must ensure that the informational quality of the data is maintained, even though its appearance will be altered. The realized record, which is qualitative, is systematized in the form of a code matrix, thus remaining qualitative; however, code matrices can be analyzed quantitatively using powerful statistical techniques. This code matrix is essential for the process of quantitizing qualitative data (Anguera, 2020, 2021; Anguera et al., 2021). Precisely, our widely explained proposal is to apply Bakeman’s (1978) initial proposal that the types of data used in systematic observation derived from their sequential or concurrent nature and the fact that they will be based on primary parameters (frequency, order and duration) (Figure 31.1), according to a progressive order of inclusion in which they gradually acquire more power (Anguera et al., 2017; Bakeman, 1978). The order parameter is the common thread of each record and allows the ordering of successive rows in the code matrices into which the descriptive record is transformed. As a result, as early as 2016, we could argue that the observational methodology is mixed methods in itself, as it integrates qualitative and quantitative elements in its own process (Anguera & Hernández-Mendo, 2016; Anguera et  al., 2017, 2020). In a broader perspective, the connecting allows the alternation of QUAL-QUAN-QUAL stages, which legitimizes the generic mixed methods approach, while a total integration between qualitative and quantitative elements is realized. With this rethinking, we can ensure an innovative way to concretize this proposal of quantitizing that will clearly materialize in systematic observation.

Research gap Between MMR and Spanish Culture: The Need for a Bridge Poth (2018a) refers to the reasons why we decided to use mixed methods, noting a growing trend in recent decades and considering various disciplines and

The lag sequential analysis, which allows the detection of stable behavioral patterns, has been proposed by Professor Bakeman (Georgia State University, USA) in collaboration with Professor Quera (University of Barcelona, Spain) (Bakeman, 1978; Bakeman & Quera, 2011) and by our group in countless applications using the SDIS-GSEQ and GSEQ5 programs. The lag sequential analysis aims to detect the existence of regularities (behavior patterns) in categorical data (Bakeman & Gottman, 1986); it can be applied diachronically in direct observation and indirect observation, including complete sessions, parts of sessions, session aggregation (Anguera et al., 2021). The polar coordinate analysis is another analytical technique, proposed by Sackett (1980) of the University of Washington (USA) and optimized by Anguera (1997) of the University of Barcelona (Spain), to identify statistically significant relationships between a behavior of interest and associated behaviors in the form of a vector map; this technique has expanded in recent years through its inclusion in the HOISAN program (Hernández-Mendo et al, 2012), and recently, through the optimization of the vector graph in R (Rodríguez-Medina et al., 2022). This analysis aims to obtain a map of interrelationships between the codes (behavior/categories) of an observational record and to represent them by means of vectors. It also requires starting from a previous record in which the occurrences/co-occurrences of codes are available sequentially (occurrences if our observation instrument had one dimension, and co-occurrences if it had several dimensions) (Anguera et al., 2021). The interpretation of these vectors is accomplished by taking into account their angle and the quadrant in which they are located, indicating the nature of the interrelation between the focal behavior and each conditioned behavior, and by considering their length, which will indicate the presence or absence of statistical significance. Furthermore, T-Patterns analysis, i.e., hidden temporal patterns of behavior, was proposed by Prof. Magnusson (University of Iceland, Rejkyavík) in close contact with us (and members of the MASI network since 1995), and the free software THEME [www.patternvision.com] has led to a very wide dissemination of this analytical technique (Magnusson, 1996, 2000, 2020).The T-Pattern analysis is a multivariate approach for the detection and description of recurring sequences of behavioral events, in order to search for hidden repeated patterns in behavior, based on a model of the temporal organization of behavior (Anguera et al., 2021).

Lag sequential analysis

Source: Author created.

Detection of T-Patterns

Polar coordinate analysis

Aim

Data analysis

THEME

HOISAN (with the addition of R graphs)

GSEQ5 HOISAN

Software

Arias-Pujol & Anguera (2020a) Santoyo et al. (2020).

Alvarado et al. (2021) Arias-Pujol & Anguera (2020a) Del Giacco et al. (2020)

Arias-Pujol & Anguera (2020a) Del Giacco et al. (2020)

Applied studies

Table 31.2  Basic quantitative data analysis techniques in systematic observation studies (with methodological contributions made from the Spanish research culture)

EXAMINING THE INFLUENCES OF SPANISH RESEARCH CULTURE 451

452

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 31.1  Data types and relation with primary parameters (frequency, order, and duration), according to Bakeman (1978) Source: Author created.

scientific journals. Its main strength is the integration of qualitative and quantitative data (Creamer, 2018), which reduces the weakness of each of them separately (Poth, 2018b). This aspect is magnificently expressed through the following words: I recognize the uniqueness of contributions of qualitative and quantitative outcomes to the integration procedures: whereas qualitative data generates text- and image-based evidence, quantitative data generates numeric-based evidence. Thus, my rationale for using mixed methods is focused on the mixing purposes addressing the problem, the integration procedures guiding the design, and the mixed insights providing access to novel contributions. (Poth, 2018a, p. 28)

In this chapter, in which we explain how systematic observation is structured as a mixed method itself, the three “ingredients” are met: mixing purposes, integration procedures and mixed insights. This justification about the use of mixed methods and the coverage they provide allows us to argue for conducting these studies in a “here” and “now” that is tied to the cultural environment in which we find ourselves and the moment in which we pose the problem; the latter is influenced and conditioned by the network of elements that make up a given cultural environment. This increased weight and relevance of cultural factors has led to assume a role in mixed methods research and in the research process. In fact, as Harris (2021) states, data collection, analysis and interpretation of the results are difficult to “disentangle” from the researchers themselves, who are immersed in their cultural guidelines and with a certain positioning. For an in-depth discussion of cultural responsiveness in mixed methods research, see Chapter 27 (this volume).

In this chapter, we reflect precisely on the importance of specific cultural factors within mixed methods (Fetters & Molina-Azorín, 2019), emphasizing how culture can influence both the design and the operationalization of mixed methods in developed countries (Fàbregues et  al., 2021). Our Spanish cultural context is framed within the European sphere and is strongly connected with the Latin American countries, with which the language unites us especially.

Purpose of the Chapter The objective of this chapter is to show how, in the Spanish context, the observational methodology has deep roots and has a highly developed and consolidated procedure that allows the integration of qualitative and quantitative elements, which is why it can be considered as mixed methods in itself. Two examples are presented that illustrate it.

ILLUSTRATIVE EXAMPLES The two examples we present below have in common a professional intervention focused on the communicative exchange, which is the substantive aspect in which both cases are located. From the mixed methods perspective and within the framework of the research culture we have referred to, we agree on the study of communicative exchanges. The analysis of this process requires the use of systematic observation and, therefore, the construction of ad hoc observation instruments, as has been done in the studies of Arias-Pujol and Anguera (2017, 2020a, 2020b) on psychotherapy with

EXAMINING THE INFLUENCES OF SPANISH RESEARCH CULTURE

adolescents, Arias-Pujol et  al. (2015) on psychotherapy with an autistic child, in those of Del Giacco et  al. (2019, 2020) on adult psychotherapy, and in that of Alvarado et al. (2021) on conflict mediation.

Example 1: Psychotherapy Objectives. The general purpose of the study is to analyze the interaction between a child with severe autism spectrum disorder ASD and his therapist. Specific objectives were: (1) to describe the behaviour of a child with severe ASD and his psychotherapist in a systematic way; (2) to identify changes in behaviour throughout the sessions; (3) to identify activation and inhibition relationships between behaviors. Methods. The observational design is nomothetic/follow-up/multidimensional (N/F/M): nomothetic because we studied the interaction between therapist and autistic child, with intersessional follow-up (because different sessions were categorized) and intrasessional follow-up (because each session was recorded in order, from the beginning to the end) and multidimensional because different dimensions were studied (Anguera et al., 2001).

453

There were two participants. The patient was a 4-year-old boy with a diagnosis of autism spectrum disorder, according to the clinical criteria of DSM-5 (APA, 2013), of the severe type according to the results obtained from the ADOS (Autism Diagnostic Observation Schedule; Lord et  al., 2000). He had no language, although he did emit sounds and some syllables forming echolalia. The therapist was a clinical psychologist with training and experience in psychoanalytic psychotherapy with children. An ad hoc observation instrument (Arias-Pujol et  al., 2015) was built as a field format modality combined with category systems. A fragment (Table 31.3) of the adaptation made by Bachs and Arias-Pujol (Bachs, 2019) is shown. This observation instrument was structured in 29 dimensions (9 of the patient, 8 of reciprocal social interaction and 12 of the therapist), and it includes a total of 57 categories. The intervention was designed for children with severe ASD and sought to stimulate reciprocal social interaction in the child through imitation in a therapeutic context (Arias-Pujol, Fieschi, Castelló, et  al., 2015; Arias-Pujol, Fieschi, & Mestres, 2015 ) (Figure 31.2). A record fragment (Bachs, 2019) is presented that is initially descriptive and hardly systematized

Figure 31.2  Graphic information about the psychotherapeutic intervention performed in Carrilet Training and Research Center (By permission of Carrilet Training and Research Center of Barcelona)

454

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 31.3  Fragment of the observation instrument (field format combined with category systems) used in psychotherapy and elaborated ad hoc. This fragment includes a small part of the categories of the therapist NAME AND DEFINITION

CODE

1.  Verbalize: Put into words. The T uses language to bring the child closer to the symbolization process (following the psycho-pedagogical scheme of Dra. Coromines, 1991; Viloca, 1998, 2003). 1.1. Describes a behavior or an object. Suggest to the child a feeling or a desire. It puts words to VD the child’s behaviors, feelings, emotions, desires, and thoughts. Given the hypersensitivity of these children, it includes naming interferences and giving information from the context such as ambient noises, if the material makes noise or if an object fails. Examples • Name object and sensation: “Oh! It’s a sponge, it’s hard and scratchy”. • Behavior: “¡Ah, you want it all for yourself!”, “You have seen the bubbles and left the dice”, “You have thrown them all”. • Desire: “Shall we blow?”, “More?”. • Context: “What a loud noise, huh? Tocotoco … they made noise”, “What noise you make, huh? You hear noise outside and you make noise, tacataca!”. 1.2. Offers help: the therapist verbalizes an offer of help to the child when it sees that he/she needs it. Example: “Let’s see, can I help you … open the box?” + opens the box. Example: “Do you want to open the box?” + pauses + “the colors?”

VOA

1.4. Anticipates: the therapist anticipates actions that will be carried [out] soon, in the immediate or distant future.

VAN

Example: “Do you want to open the box?” + pauses + “The colors?” Example: The therapist says: “At one, at two and at … “singing + inhale air + blow the bubbles. Example: The child takes the cup and puts it in his/her mouth. Meanwhile, the therapist says: “Let’s put some water” + takes another cup from the box. Source: Excerpted from Bachs (2019) With permission from Bachs.

compared to another already coded by the observation instrument (Table 31.4). The interobserver agreement was found, obtaining a kappa coefficient = .94 (Cohen, 1960). Data quality control was performed using the free program GSEQ. The quantitative analysis carried out from the data collected is very extensive, considering that there are 57 codes in the observation instrument. For this reason, we will focus only on the analysis of focal coordinates due to its robustness and ability to obtain an interrelational map between the categories. Therefore, 57 polar coordinate analysis would derive from the constructed observation instrument, considering each of the categories as a focal behavior. The HOISAN program (Hernández-Mendo et al., 2012) has been used to perform the obtained parameters (Table 31.5) and the R program (Rodríguez-Medina et al., 2022) for vector optimization. Results. At the end of the treatment, the pre-post comparison of the ADOS test did not detect significant changes in the degree of severity of autism, but it did detect significant changes in some subitems of the language and communication domain test. The systematized description of the behaviour of the child through the ad hoc elaborated instrument

showed us that, despite being a child who spent most of the time disconnected (70 per cent), the behaviours of the dimension “reciprocal social interaction” increased from 30 per cent to 48 per cent. All this means that, during treatment, the child has developed the capacity to ask, follow a suggestion, approach the therapist, make physical contact and eye contact, and respond to imitation. The analysis by polar coordinates allows us to continue studying in detail the activation and inhibition relation between these behaviors and capture aspects of the child–therapist interaction that would go unnoticed without mathematical analysis (Anguera, 1997; Sackett, 1980). Specifically, we are interested in elucidating the impact of the therapist’s verbal and non-verbal behaviour on the child.

INTEGRATION PROCEDURES AND INSIGHTS The child studied presented a severe ASD with very change-resistant behaviours, poor eye contact,

455

EXAMINING THE INFLUENCES OF SPANISH RESEARCH CULTURE

Table 31.4  Record fragment where the initial transcription is shown, barely systematized and the coding to which it gives rise, thanks to the observation instrument. The + sign indicates co-occurrence, both for the therapist and for the patient TRANSCRIPT

CODING

OBSERVER 1

Therapist

Therapist

Therapist

Patient

(Outside the set) – “Do we enter? … We have already arrived …”

Enters running (surrounding the table) + gestures with his hands and stops when he reaches the door + turns + looks at T + smiles + looks sideways at the camera.

VD

EM + EM + AN + CO + EXFA + EVI

(Outside the set) – Laughs + entering gestures with his hands and stops when he reaches the door + looks at the child + “Very well"

He/she goes to the table and stands behind the chair + and touches it with both hands + smiles

R + INVT + MI + VSUP

LLU + AS + EXFA

Looks at the child (from the door)

Moves his head from side to side + reviews the outline of the chair with one hand + looks at T + smiles + walks to the other side (not the door)

MI

EM + AS + CO + EXFA + LLU

Source: Extracted from Bachs (2019). With permission from Bachs.

Table 31.5  Table of parameters corresponding to the analysis of polar coordinate, with VSUP as focal behaviour and all the others as conditional. Only the conditioned behaviours that generate significant (*) and very significant (**) vectors have been selected Category Stereotypies_EM Stereotypies_EVI Autosensory_AS Response to a demand_NFA Proxemic behavior_APRO Proxemic_behavior_LLU Physical_contact_CCB Facial_expresion_EXFA Imitates_IVT Proxemic_behavior_ therapist_APROT Proxemic_behavior_ therapist_LLUT Source: Author created.

Quadrant

Prospective Retrospective Ratio Zsum Zsum

Length

Significance

Angle

I IV I I I I I I II IV

4,13 3,19 2,14 1,11 1,57 2,8 0,79 4,89 −0,79 2,58

1,6 −0,71 0,55 3,38 1,85 0,46 4,27 1,37 2,36 −0,31

0,36 −0,22 0,25 0,95 0,76 0,16 0,98 0,27 0,95 −0,12

4,43 3,27 2,21 3,56 2,42 2,83 4,34 5,08 2,49 2,6

** ** * ** * ** ** ** * **

21,18 347,54 14,4 71,85 49,67 9,26 79,57 15,68 108,57 353,11

II

−0,5

1,97

0,97

2,03

*

104,19

456

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

motor, visual and verbal stereotypies, auditory hypersensitivity and great interaction difficulties. These behaviours changed throughout the sessions; little by little, he allowed the therapist to approach him, physical contact also appeared, his ability to say some words, make sounds, hum and laugh also changed; between them, a sort of proto-conversation appeared (in the sense of Trevarthen, 2005), in which, sounds and words alternated with movements and facial expressions

in a game of turns, establishing a certain emotional connection between therapist and child. His gaze went from being elusive to being directed more and more towards the eyes and the facial contour. To objectify these changes that are observed qualitatively from clinical practice, it is necessary to develop an observational instrument and follow the steps of the observational methodology from the mixed methods perspective (Anguera et al., 2018).

Table 31.6.  Fragment of the observation instrument (field format modality) elaborated ad hoc Context

Caller

Verbal dimension

Non-verbal dimension: Facial expression

Cod

Space

Cod

Who

Cod

Behaviors

Cod

Face

A1

Domic.part 1

B1

Part 1

C1

Questions

D1

Firmness

A2

Domic.part 2

B2

Part 2

C2

Complains

D2

Reflection

A3

Exterior 1

B3

Lawyer. P 1

C3

Exclaims

D3

Sadness

A4

Mediator’s office

B4

Lawyer. P2

C4

Responds

D4

Disbelief

A5

Lawyer’s office 1

B5

Med. 1

C5

Arguments

D5

Concentration

A6

Lawyers office 2

B6

Med. 2

C6

Demands

D6

Anger

Source: Author created.

Examples

EXAMINING THE INFLUENCES OF SPANISH RESEARCH CULTURE

Example 2: Mediation in conflicts Objectives. The case presented corresponds to a mediation in a family and intergenerational conflict between a mother-in-law and her daughter-inlaw (widow). This information is public because it is on YouTube (www.youtube.com/watch?v= jzwAgdNNOoo). The objective is to detect if there are optimizing patterns in the communicative exchange, initially very negative, that allow verifying the effectiveness of the mediation, resolving the conflict and reaching an agreement between grandmother and daughter-in-law. Methods. The observational design is nomothetic/follow-up/multidimensional (N/F/M): nomothetic, since there are several participants, intersessional (four sessions) and intrasessional follow-up (follow-up in each record from the beginning to the end of the session), and multidimensional, since various dimensions were proposed in the observation instrument (SánchezAlgarra & Anguera, 2013). The participants are the two people in conflict (mother-in-law and daughter-in-law), as well as two lawyers (one from each party), and the two mediators who cooperatively led the sessions. We built an ad hoc observation instrument, which belongs to field format modality, since several dimensions were proposed, which were displayed. A fragment of the observation instrument is presented in Table 31.6, which corresponds to different subdimensions of non-verbal and verbal behavior. Previous sessions of each party were recorded at home, the consultation sessions of each party with their respective lawyers were recorded in the lawyer’s office, and the mediation sessions in the mediator’s office. All sessions were coded through the free program LINCE. The record was carried out using the free LINCE program, obtaining the corresponding code matrices, which constitute a systematized record. Interobserver agreement was found between three records, which is recommended when some codes present a certain risk of subjectivity, as could occur with facial expressions. Through the free HOISAN program, the canonical agreement coefficient of Krippendorff (2018) was found. Data analysis began with the record obtained through the LINCE program and was exported to the free program THEME in order to achieve the detection of T-patterns, as one of the robust techniques for quantitative analysis of observational data. The detection of T-patterns works thanks to a powerful algorithm that allowed us to reveal behavioral configurations in each of the mediation sessions. Results. As the recording and the corresponding analysis was carried out separately in each of the four sessions, the results show at the beginning the frontal and hard confrontation between the two

457

parties, with attempts to overlap turns, contemptuous phrases, high voice level and sometimes disjointed faces, in T-patterns of such a level of complexity. The evolution of the T-patterns in the successive sessions shows the satisfactory progress that took place in the resolution of this conflict. Integration procedures and insights. Conflict mediation is an area in which the intervention has great possibilities for the analysis of verbal and non-verbal behaviour through systematic observation. Expressiveness constitutes an enormously complex area, in an interplay between the subdimensions of verbal behavior and non-verbal behaviour, and that is fully manifested in this illustrative situation chosen as example 2. This information is qualitative, but having used the order parameter (Bakeman, 1978) in the systematized record, and therefore meeting the requirements for a diachronic analysis such as the detection of T-patterns (Anguera et  al., 2021; Magnusson, 1996, 2000, 2020; Santoyo et  al., 2020), which is quantitative and a mathematically very robust resource, it is possible to obtain objective results that are the result of this intended integration in mixed methods (Bazeley, 2018). Consequently, it allows us to affirm that the objective has been met.

Challenges Encountered in the Illustrative Examples Influence by a global research culture. These challenges have focused on the conceptual progress of mixed methods, increasingly consolidated precisely in the global research culture and in various international organizations (Fetters & MolinaAzorín, 2021). The qualitative and quantitative research traditions constitute different cultures, marked by different values, beliefs and norms (Mahoney & Goertz, 2006; Onwuegbuzie et  al., 2010), which have materialized in diverse areas and which they describe as the amalgamation of two cultures or traditions. The expansion of mixed methods that is taking place worldwide and not only in developed countries—although mainly in them—generates not only challenges, but also new possibilities for progress (Harris, 2021), and especially regarding integration (Creamer, 2018). And this culture of integration has been a crucial challenge in these two illustrative examples. In both we have achieved the desired integration between qualitative and quantitative elements, overcoming the difficulty involved in each of them. And in both we value as especially useful and effective the use of the observational methodology. Research in resource-poor settings. The studies that are carried out using observational

458

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

methodology do not require financially expensive resources, but only to be able to have recorded sessions of the situation studied, a requirement that has now become enormously cheaper. It is enough to bear in mind today that most of the computer programs for recording, data quality control, and analysis, to which we have previously referred (and, specifically, GSEQ, HOISAN, LINCE, LINCE PLUS), are free and therefore allow universal access. Consequently, researchers and professionals with few resources can carry out research similar to what we have described. Challenges of cross-cultural collaboration. Bedwell et al. (2012) define collaboration as “an evolving process whereby two or more social entities actively and reciprocally engage in joint activities aimed at achieving at least one shared goal” (p. 130). Effective collaboration always benefits the resulting product and currently a widespread interest exists in cross-cultural collaboration, not always easy to define, but which is facilitated by advances in communication and that must overcome various barriers. Some of these barriers are as basic as the use of similar terms defined differently or the lack of differentiation between similar constructs (Bedwell et al., 2012). Most of the literature considers collaboration as a process (Bedwell et al., 2012) in which successive “members” adhere in search of desirable common results and in a continuous process of involvement (Schadewitz, 2009). In order for a cross-cultural collaboration to take place properly, an active process is required that progresses over time and in which the relationship between different “research units” takes shape, although the peculiarity of each discipline plays a fundamental role (Bedwell et al., 2012). From our point of view, progressive cross-cultural collaboration in the field of systematic observation is perfectly feasible and we intend to contribute our grain of sand in this regard.

IMPLICATIONS FOR OTHER GLOBAL RESEARCHERS FOR USING THIS MMR DESIGN Continuing with this approach, it is clear that collaboration is more than the sum of its parts, and the clarity of roles and responsibilities positively influences collaboration with other global researchers with a collaborative will (Gil-García et al., 2019). It is necessary to previously analyze and reflect on the most suitable collaboration structure, assessing the implications of all the participating researchers.

The implications that we consider necessary for other global researchers for using this MMR design have a background of mental availability open to innovation in MMR (Poth, 2018a), which demands creative designs, making the phrase “we have to get beyond the ‘I can’t because …’ thoughts and instead see the possibilities framed with ‘but I could if ‘ … ’ (p. 291). As Poth (2018a) states, research on mixed methods in complex situations requires “embedding creativity in our everyday practices” (p. 291), and precisely systematic observation is characterized by the study of habitual behaviour, which, in our illustrative examples, have informed the psychotherapist and conflict mediators in their daily professional activities.

CONCLUSIONS As an epilogue to this work, we value the effectiveness that the systematic observation—which we have illustrated here in two examples related to communication applied to the fields of psychotherapy and conflict mediation—contributes to the study of the research from the perspective of mixed methods, and taking into account the experience accumulated in the last four decades, while there have been methodological and technological advances that assess its great possibilities. The range of possibilities that opens up is extraordinarily wide, and the ideal methodological approach must be adapted to each case, which is possible within the culture of research. The challenge is very important for a sustained expansion of collaboration with other researchers, in an increasingly global framework, given the possibilities of access to free resources that facilitate progressive involvement in a research culture.

WHAT TO READ NEXT We hope that the readers of this chapter will be encouraged to undertake an observational study within the framework of mixed methods. Anguera, M. T., Blanco-Villaseñor, A, Losada J. L., & Sánchez-Algarra, P. (2020). Integración de elementos cualitativos y cuantitativos en metodología observacional [Integration of qualitative and quantitative elements in observational methodology]. Ámbitos. Revista Internacional de Comunicación,

EXAMINING THE INFLUENCES OF SPANISH RESEARCH CULTURE

49, 49–70. https://doi.org/10.12795/Ambitos. 2020.i49.04

This article demonstrates rigour and flexibility as essential in observational methodology. There is a proposal about observational methodology considered itself as mixed methods, that imply integration ways (quantitizing) between qualitative and quantitative elements. Anguera, M.T., Portell, M., Chacón-Moscoso, S., & Sanduvete-Chaves, S. (2018). Indirect observation in everyday contexts: Concepts and methodological guidelines within a mixed methods framework. Frontiers in Psychology, 9(13). https://doi. org/10.3389/fpsyg.2018.00013

This article describes indirect observation as involving analyzing textual material generated either indirectly from transcriptions of audio recordings of verbal behaviour in natural settings. Narratives are an excellent vehicle for studying everyday life, and quantitization is proposed as a means of integrating qualitative and quantitative elements. There is a methodological framework detailing the steps and decisions required to quantitatively analyze a set of data that was originally qualitative. Anguera, M. T., Portell, P., Hernández-Mendo, A., Sánchez-Algarra, P., & Jonsson, G. K. (2021). Diachronic analysis of qualitative data. In A.J. Onwuegbuzie and B. Johnson (Eds.), Reviewer’s guide for mixed methods research analysis (pp. 125–138). Routledge. https://doi. org/10.4324/9780203729434-12

This article explains three techniques of diachronic quantitative analysis that apply to qualitative data: the lag sequential analysis, the analysis of polar coordinates and the TPA offer three different ways for a diachronic analysis, always involving scrutinizing the structure of behaviour and from rigorous quantitative analyses that are very suitable for qualitative data. The suggested analysis can help mixed method researchers both design their mixed analysis as well as analyze their data coherently.

ACKNOWLEDGEMENTS The authors gratefully acknowledge the support of a Spanish government subproject, Integration ways between qualitative and quantitative data, multiple case development and synthesis review as the main axis for an innovative future in physical activity and sports research [PGC2018-098742B-C31] (2019–2021) (Ministerio de Ciencia, Innovación y Universidades/Fondo Europeo de

459

Desarrollo Regional) that is part of the coordinated project, New approach of research in physical activity and sport from mixed methods perspective [SPGC201800X098742CV0]. In addition, the first author gives thanks for the support of the Spanish government project Integration between observational data and data from external sensors: Evolution of the LINCE PLUS software and development of the mobile application for the optimization of sport and health-enhancing physical activity (EXP_74847) (2023), Spanish Ministry of Culture and Sports, Consejo Superior de Deportes, and the European Union. Also, the first author thanks the Generalitat de Catalunya Research Group and Innovation in Designs (GRID). Technology and multimedia and digital application to observational designs [2021 SGR 00718] (2022-2024). Moreover, the research that has led to these results has been carried out through the funds of the Fundació la Caixa.

REFERENCES Alvarado, C., Armadans, I., Parada, M. J., & Anguera, M. T. (2021). Unraveling the role of shared vision and trust in constructive conflict management of family firms. An empirical study from a mixed-methods approach. Frontiers in Psychology, 12: 629730. https://doi.org/10.3389/fpsyg.2021.629730 Anguera, M. T. (1979) Observational typology. Quality & Quantity. European-American Journal of Methodology, 13(6), 449–484. https://doi. org/10.1007/bf00222999 Anguera, M. T. (1997). From prospective patterns in behavior to joint analysis with a retrospective perspective. In Proceedings of the Colloque sur invitation “Méthodologie d’analyse des Interactions Sociales”. Université de la Sorbonne. Anguera, M. T. (2003). Observational methods (general). In R. Fernández-Ballesteros (Ed.), Encyclopedia of Psychological Assessment (Vol. 2, pp. 632–637). Sage. Anguera, M. T. (2020). Is it possible to perform ‘liquefying’ actions in conversational analysis? The detection of structures in indirect observation. In L. Hunyadi & I. Szekrényes (Eds.), Intelligent systems reference library: Vol. 164. The temporal structure of multimodal communication: Theory, methods and applications (pp. 45–67). Springer. https://doi. org/10.1007/978-3-030-22895-8_3 Anguera, M. T. (2021). Desarrollando la observación indirecta: Alcance, proceso, y habilidades metodológicas en el análisis de textos [Developing indirect observation: Scope, process, and methodological skills in text analysis]. In C. Santoyo &

460

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

L. Colmenares (Eds.), Patrones de habilidades metodológicas y conceptuales de análisis, planeación, evaluación e intervención en Ciencias de la Conducta (pp. 189–217). UNAM/PAPIIT, IN306715. Anguera, M.T., Blanco-Villaseñor, A. y Losada, J.L. (2001). Diseños Observacionales, cuestión clave en el proceso de la metodología observacional [Observational designs, a key question in the observational methodology process]. Metodología de las Ciencias del Comportamiento, 3(2), 135–161. Anguera, M. T., Blanco-Villaseñor, A, Losada J. L., & Sánchez-Algarra, P. (2020). Integración de elementos cualitativos y cuantitativos en metodología observacional [Integration of qualitative and quantitative elements in observational methodology]. Ámbitos. Revista Internacional de Comunicación, 49, 49–70. https://doi.org/10.12795/Ambitos. 2020.i49.04 Anguera, M. T., Blanco-Villaseñor, A., Losada, J. L., Sánchez-Algarra, P., & Onwuegbuzie, A. J. (2018). Revisiting the difference between mixed methods and multimethods: Is it all in the name? Quality & Quantity, 52(6), 2757–2770. https://doi.org/ 10.1007/s11135-018-0700-2 Anguera, M. T., Camerino, O., Castañer, M., SánchezAlgarra, P., & Onwuegbuzie, A. J. (2017). The specificity of observational studies in physical activity and sports sciences: Moving forward in mixed methods research and proposals for achieving quantitative and qualitative symmetry. Frontiers in Psychology, 8. https://doi.org/10.3389/ fpsyg.2017.02196 Anguera, M. T., & Hernández-Mendo, A. (2016). Avances en estudios observacionales en ciencias del deporte desde los mixed methods [Advances in mixed methods observational studies in sports sciences]. Cuadernos de Psicología del Deporte, 16(1), 17–30. https://doi.org/10.4321/S157884232015000100002 Anguera, M. T., & Izquierdo, C. (2006). Methodological approaches in human communication. From complexity of situation to data analysis. In G. Riva, M. T. Anguera, H. Wiederhold, & F. Mantovani (Eds.), From communication to presence: Cognition, emotions and culture towards the ultimate communicative experience (pp. 207– 225). IOS Press. Anguera, M. T., Magnusson, M. S., & Jonsson, G. K. (2007). Instrumentos no estándar [Non-standard instruments]. Avances en Medición, 5(1), 63–82. Anguera, M. T., Portell, M., Chacón-Moscoso, S., & Sanduvete-Chaves, S. (2018). Indirect observation in everyday contexts: Concepts and methodological guidelines within a mixed methods framework. Frontiers in Psychology, 9. https://doi.org/10.3389/ fpsyg.2018.00013

Anguera, M. T., Portell, P., Hernández-Mendo, A., Sánchez-Algarra, P., & Jonsson, G. K. (2021). Diachronic analysis of qualitative data. In A. J. Onwuegbuzie & B. Johnson (Eds.), Reviewer’s guide for mixed methods research analysis (pp. 125–138). Routledge. https://doi-org/ 10.4324/ 9780203729434-12 American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author. Arias-Pujol, E., & Anguera, M. T. (2017). Observation of interactions in adolescent group therapy: A mixed methods study. Frontiers in Psychology, 8:1188. https://doi.org/10.3389/fpsyg.2017.01188 Arias-Pujol, E., & Anguera, M. T. (2020a). A mixed methods framework for psychoanalytic group therapy: From qualitative records to a quantitative approach using T-Pattern, lag sequential and polar coordinate analyses. Frontiers in Psychology, 11: 1922. https://doi.org/10.3389/fpsyg.2020.01922 Arias-Pujol, E., & Anguera, M. T. (2020b). Investigación en psicoterapia psicoanalítica: metodología observacional desde la perspectiva mixed-methods [Research in psychoanalytic psychotherapy: observational methodology from the perspective mixedmethods]. In J. A. Castillo-Garayoa & J. Mercadal Rotger (Eds.), Psicoterapia psicoanalítica. Investigación, evaluación y práctica clínica (pp. 225– 249). Herder. Arias-Pujol, E., Fieschi, E., Castelló, J., Miralbell, A., Soldevila, E., Sánchez-Caroz, E., Anguera, M. T., & Mestres, M. (2015). Efectos de la imitación en la interacción social recíproca en un niño con trastorno del espectro autista grave [Effects of imitation on mutual social interaction in children with severe autism spectrum disorder]. Revista de Psicopatología y Salud Mental del Niño y del Adolescente, 25, 9–20. Arias-Pujol, E., Fieschi, E., & Mestres, M. (2015). La imitació del nen autista en la psicoteràpia psicoanalítica: fonaments, disseny i aplicación [The imitation of the autistic child in psychoanalytic psychotherapy: fonaments, design and application]. Revista Catalana de Psicoanàlisi, 32(1), 123–37. https://www.raco.cat/index.php/RCP/article/view/ 299234. Bachs, N. (2019). Autisme infantil: adaptació i validació d’un instrument observacional [Child autism: adaptation and validation of an observational instrument] [Unpublished bachelor’s thesis]. Universitat Ramon Llull. Bakeman, R. (1978). Untangling streams of behavior: Sequential analysis of observation data. In G. P. Sackett (Ed.), Observing behavior: Vol. 2. Data collection and analysis methods (pp. 63–78). University Park Press. Bazeley, P. (2018). Integrating analyses in mixed methods research. Sage.

EXAMINING THE INFLUENCES OF SPANISH RESEARCH CULTURE

Bedwell, W. L., Wildman, J. L., Diaz Granados, D., Salazar, M., Kramer, W. S., & Salas, E. (2012). Collaboration at work: An integrative multilevel conceptualization. Human Resource Management Review, 22(2), 128–145. https://doi.org/10.1016/j. hrmr.2011.11.007 Blanco-Villaseñor, A., Castellano, J., HernándezMendo, A., Sánchez-López, C.R., & Usabiaga, O. (2014). Aplicación de la TG en el deporte para el estudio de la fiabilidad, validez y estimación de la muestra [Application of the generalizability theory in sport to study the validity, reliability and estimation of samples]. Revista de Psicología del Deporte, 23(1), 131–137. Blanco-Villaseñor, A., Sastre i Riba, S., & EscolanoPérez, E. (2010). Desarrollo ejecutivo temprano y Teoría de la Generalizabilidad: Bebés típicos y prematuros [Executive function in early childhood and Generalizability Theory: Typical babies and preterm babies]. Psicothema, 22(2), 221–226. Chacón-Moscoso, A., Anguera, M. T., SanduveteChaves, S., Losada, J. L., Lozano-Lozano, J. A., & Portell, M. (2019). Methodological quality checklist for studies based on observational methodology (MQCOM). Psicothema, 31(4), 458–464. https://doi.org/10.7334/psicothema2019.116 Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. https://doi.org/10.1177/ 001316446002000104 Creamer, E. G. (2018). An introduction to fully integrated mixed method research. Sage. Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (2nd ed., 2011). Sage. Del Giacco, L., Anguera, M. T., & Salcuni, S. (2020). The action of verbal and non-verbal communication in the therapeutic alliance construction: A mixed methods approach to assess the initial interactions with depressed patients. Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg.2020.00234 Del Giacco, L., Salcuni, S., & Anguera, M. T. (2019). The Communicative Modes Analysis System in Psychotherapy from mixed methods framework: Introducing a new observation system for classifying verbal and nonverbal communication. Frontiers in Psychology, 10. https://doi.org/10.3389/ fpsyg.2019.00782 Fàbregues, S., Molina-Azorín, J. F., & Fetters, M. D. (2021). Virtual special issue on “Quality in Mixed Methods Research”. Journal of Mixed Methods Research, 15(2), 146–151. https://doi. org/10.1177/15586898211001974 Fetters, M. D., & Molina-Azorín, J. F. (2019). A call for expanding philosophical perspectives to create a more “worldly” field of mixed methods: The example of Yinyang philosophy. Journal of Mixed

461

Methods Research, 13(1), 15–18. https://doi.org/ 10.1177/1558689818816886 Fetters, M. D., & Molina-Azorin, J. F. (2021). Guidance on using mixed methods from diverse international organizations in the behavioral, social, fundamental, and health sciences. Journal of Mixed Methods Research, 15(4), 470–484. Gil-García, J. R., Guler, A., Pardo, T. A., & Burke, G. B. (2019). Characterizing the importance of clarity of roles and responsibilities in government interorganizational collaboration and information sharing initiatives. Government Information Quarterly, 36, 101393. Hanover Research (2012). Building a culture research: Recommended practices. Academy Administration Practice. Harris, J. (2021). Mixed methods research in developing country contexts: Lessons from field research in six countries across Africa and the Caribbean. Journal of Mixed Methods Research. https://doi. org/10.1177/15586898211032825 Hernández-Mendo, A., López-López, J. A., Caste­ llano, J., Morales-Sánchez, V., and Pastrana, J. L. (2012). Hoisan 1.2: Programa informático para uso en metodología observacional [Hoisan 1.2: software for observational methodology]. Cuadernos de Psicología del Deporte, 12, 55-78. https://doi.org/10.4321/S1578-84232012000100006 Izquierdo, C. & Anguera, M. T. (2021). The analysis of interpersonal communication in sport from mixed methods strategy: The integration of qualitativequantitative elements using systematic observation. Frontiers in Psychology, 12:637304. https:// doi.org/10.3389/fpsyg.2021.637304 Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi. org/10.1177/1558689806298224 Krippendorff, K. (2018). Content analysis: An introduction to its methodology. Sage. Lord, C., Risi, S., Lambrecht, L., Cook, E., Leventhal, B., Dilavore, P. C., Pickles, A., & Rutter. M. (2000). The autism diagnostic observation schedule – generic: a standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223. https://doi.org/10.1023/ A:1005592401947 Losada, J. L., & Manolov, R. (2015). The process of basic training, applied training, maintaining the performance of an observer. Quality & Quantity, 49(1), 339–347. https://doi.org/10.1007/s11135014-9989-7 Magnusson, M. S. (1996). Hidden real-time patterns in intra- and inter-individual behavior. European Journal of Psychological Assessment, 12(2), 112– 123. https://doi.org/10.1027/1015-759.12.2.112

462

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Magnusson, M. S. (2000). Discovering hidden time patterns in behavior: T-patterns and their detection. Behavior Research Methods, Instruments, and Computers, 32(1), 93–110. https://doi.org/10.3758/ BF03200792 Magnusson, M. S. (2020). T-pattern detection and analysis (TPA) with THEMETM: a mixed methods approach. Frontiers of Psychology, 10. https://doi. org/ 10.3389/fpsyg.2019.02663 Mahoney, J., & Goertz, G. (2006). A tale of two cultures: Contrasting quantitative and qualitative research. Political Analysis, 14(3), 227–249. https://doi.org/10.1093/pan/mpj017 O’Cathain, A. (2010). Assessing the quality of mixed methods research: Toward a comprehensive framework. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social & behavioral research (pp. 531–556). Sage. https:// doi.org/10.4135/9781506335193 Onwuegbuzie, A. J., Bustamante, R. M., & Nelson, J. A. (2010). Mixed research as a tool for developing quantitative instruments. Journal of Mixed Methods Research, 4(1), 56–78. https://doi.org/10.1177/ 1558689809355805 Portell, M., Anguera, M. T., Chacón-Moscoso, S., & Sanduvete-Chaves, S. (2015). Guidelines for reporting evaluations based on observational methodology (GREOM). Psicothema, 27(3), 283–289. Poth, C. N. (2018a). Innovation in mixed methods research. A practical guide to integrative thinking with complexity. Sage. Poth, C. N. (2018b). The curious case of complexity: Implications for mixed methods research practices. International Journal of Multiple Research Approaches, 10(1), 403–411. https://doi.org/ 10.29034/ijmra.v10n1a27

Rodríguez-Medina, J., Arias, V., Arias, B., Hernández-Mendo, A., & Anguera, M. T. (2022). Polar Coordinate Analysis, from HOISAN to R: A Tutorial Paper [Unpublished manuscript] https://jairodmed.shinyapps.io/HOISAN_to_R_2022/ Sackett, G. P. (1980) Lag sequential analysis as a data reduction technique in social interaction research. In D. B. Sawin, R. C. Hawkins, L. O. Walker, & J. H. Penticuff (Eds.), Exceptional infant. Psychosocial risks in infant-environment transactions (pp. 300– 340). Brunner/Mazel. Sánchez-Algarra, P., & Anguera, M. T. (2013). Qualitative/quantitative integration in the inductive observational study of interactive behaviour: Impact of recording and coding predominating perspectives. Quality & Quantity. International Journal of Methodology, 47(2), 1237–1257. https://doi.org/10.1007/ s11135-012-9764-6 Sandelowski, M., Voils, C. I., & Knafl, G. (2009). On quantitizing. Journal of Mixed Methods Research, 3, 208–222. Santoyo, C., Jonsson, G. K., Anguera, M. T., Portell, M., Allegro, A., Colmenares, L., & Torres, G. Y. (2020). T-Patterns integration strategy in a longitudinal study: A multiple case analysis. Physiology & Behavior, 222, 112904. https://doi.org/10.1016/j. physbeh.2020.112904 Schadewitz. N. (2009). Design patterns for crosscultural collaboration. International Journal of Design, 3(3), 37–53. Trevarthen, C. (2005). First things first: infants make good use of the sympathetic rhythm of imitation, without reason or language. Journal of Child Psychotherapy, 31(1), 91–113. https://doi.org/10.1080/ 00754170500079651

Future Direction for Navigating Research Cultures in Designs: Section 5 Conclusions Elsa Lucia Escalante-Barrios

INTRODUCTION Section 5 of the Handbook presents evidence of the expansion beyond traditional United States (US) and United Kingdom (UK) perspectives in the global growth of mixed methods research (MMR) by compiling five empirical contributions of scholars from different latitudes with diverse cultural contexts. This section revealed that (a) the MMR field is moving towards a new stage where complex designs and their features can be determined by the unique characteristics of each cultural group, and (b) there is a need to make visible the ongoing methodological dialogues among researchers and academic communities from different fields and regions. These dialogues can prompt other scholars to consider, develop and consolidate new ways of implementing MMR while taking into account and respecting the cultural components inherent to each context. As Section Leads, Dr. Escalante and Dr. Creamer compiled the efforts of the contributors to Section 5 to address specific researchable problems, which required rethinking the existing MMR designs (Chapter 31, Anguera et  al. and Chapter 30, Chu et  al.) and developing sophisticated, culturally responsive designs and approaches to collect, analyze and integrate

quantitative and qualitative data in diverse cultural contexts. Examples of the latter include designs which consider culture when engaging participants in the data collection and analysis stages of a community-based participatory MMR research project (Chapter 29, Chandanabhumma et al.), the use of a “rhetorical-analytical framework” unique to Japan (Chapter 28, Hatta), and the application of critical race theory to engineering in the United States to inform data analysis in MMR studies (Chapter 27, Hall & Boyce).

CULTURAL REFLEXIVITY IN MIXED METHODS RESEARCH Recent contributions to the MMR field have considered the significance of the broader social, environmental and cultural contexts in which MMR studies are conducted, as well as the ways in which these contexts can affect the participants’ values, attitudes and behaviours (Creswell & Sinley, 2017; Plano Clark & Ivankova, 2016; Harris, 2022). According to Helman (1994), culture is the “collection of guidelines (both explicit and implicit) that individuals acquire as members of a specific society and that instruct them on how to view the world,

464

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

how to experience it emotionally, and how to behave in relation to it” (pp. 2–3). Similarly, Nastasi, Moore and Varjas (2004) define culture as contexts or groups that share similar “ideas, beliefs, values, and behavioral norms” (cited in Nastasi & Hitchcock, 2016, p. 18) Section 5 was significantly influenced by the adoption of a cultural reflexivity lens, based on the idea that MMR researchers must pay special attention to their distinct cultural beliefs and values. By developing a better understanding of their culture, researchers can gain a more precise understanding of the cultural elements in their social environment. The types of MMR designs and analytical approaches used in this section, as well as the strategies by which they were implemented and disseminated, required researchers to consider (a) their personal experiences as members of each culture, and (b) the interactions with the social structure of their cultural environment and academic communities when making methodological decisions. For instance, in the chapter by Hall & Boyce, the authors explicitly mention their ethnicity, gender and academic interests to situate themselves within the study’s framework. By doing so, the authors could report transparently on how their cultural beliefs affected their decisions and understandings as researchers during the design and conduct of their MMR study. In this Conclusion to Section 5, we outline several future directions to ensure that the use of such a culturally reflexive approach, which is currently uncommon in MMR practice, makes a significant contribution to the MMR field. These future directions are described in detail below.

FUTURE DIRECTIONS Promoting Methodological Dialogues Between and Within Cultures to Enrich the Mixed Methods Field The chapters in this section describe MMR developments rooted in the cultural particularities of various latitudes. An essential first next step for the MMR field would be to keep bringing attention to new approaches from across the globe and, as a result, to bring together MMR communities at the local, national, and international levels. As shown in the historical overview of MMR by Creswell and Sinley (2017), while authors from Asia and Europe (excluding the United Kingdom) have made significant contributions to the field, primarily in the form of book chapters and methodological articles, authors from the United

States, the United Kingdom and the Common­ wealth countries predominate in the books and journal articles on MMR. A relatively recent initiative to overcome this limitation has been the expansion of international partnerships within the Mixed Methods International Association (MMIRA), led by members from specific geographic locations and frequently involving local scholars alone or in conjunction with researchers from other locations. Examples of these partnerships include the Japan Society for Mixed Methods Research (JSMMR), the MMIRA-CC Caribbean Chapter, the MMIRA Oceania Chapter, Méthodes Mixtes Francophonie, and the Latin American Association for Mixed Methods Research (ALIMM). With the organization of conferences and workshops on MMR, the dissemination of MMR knowledge among local researchers not familiar with the methodology and the formation of communities of practice with shared cultural research interests and practices, these organizations have contributed to the development of a more global MMR community, decentring the prevalent US- or UK-based framing of MMR, and empowering new leaderships and innovative methodological approaches.

Acknowledging How Culture Shapes Mixed Methods Thinking and Practice A second critical next step would be promoting initiatives to enhance awareness about the particularities of MMR in diverse cultural contexts and communities. This awareness is necessary to (a) train local researchers on how to use MMR to improve the study of unique cultural concerns and (b) recognize the academic cultures of countries and reflect on how cultural factors influence the conceptualization and implementation of MMR designs (Harris, 2022). Due to the diversity of research cultures, MMR researchers occasionally disagree on fundamental issues such as how MMR should be defined, what nomenclature should be used to refer to MMR concepts and procedures, and whether there is a need for consensus in MMR standpoints (Fàbregues et al., 2021). This diversity of opinion is in line with the notion of communities of practice (Denscombe, 2008), which conceptualizes MMR as a methodology comprising multiple research communities shaped by the prevailing principles, values and interests of their disciplines, research orientations, and research cultures. Instead, we argue that the variety of perspectives in the MMR field indicates that “reality is likely plural” (p. 204) rather than posing a problem (Johnson, 2009). In the future, the following steps

SECTION 5 CONCLUSIONS

465

should be taken to promote diversity in the field: (a) increase awareness of the heterogeneity perspectives on MMR; (b) promote the healthy coexistence of divergent viewpoints within the MMR community (i.e., the Journal of Mixed Methods Research is an example of such a healthy coexistence); and (c) prepare for potential challenges that may arise when working with MMR researchers from other cultures. Some of these initiatives have already materialized within the aforementioned international partnerships through actions such as the organization of MMR Regional Conferences (e.g., JSMMR Annual Conference, Australasian Pacific MMIRA Regional Conference), the organization of training workshops, and the launch of journals exclusively devoted to publishing empirical and methodological articles on MMR, written by regional authors or highlighting the role of culture in the design and conduct of MMR studies (e.g., Annals of Mixed Methods Research, Caribbean Journal of Mixed Methods Research).

journals in researchers’ native languages would be a crucial fourth step for advancing cultural sensitivity in the field of MMR. Several authors have argued that since language is context-dependent, researchers from various cultures may interpret translated concepts differently (Angel, 2013; Escalante-Barrios et al., 2020, 2021). A few sociocultural nuanced aspects of some statements might also be lost in translation. Consequently, the ability of these activities to preserve local language may thus ensure that MMR researchers can maintain the context and contextual meanings crucial to the participants’ experiences. This initiative would act as a call to action for academics worldwide to support outstanding MMR practices in their countries to circumvent the challenges posed by translationrelated issues. In addition, it would give universities, research organizations and associations across the globe a chance to promote best practices in planning and executing culturally sensitive MMR when researching complex global problems.

Making Cultural Reflexivity a Core Competency for Mixed Methods Researchers

REFERENCES

A third next step would involve encouraging awareness and assimilation of cultural reflexivity in MMR, which is crucial to enhancing cultural responsiveness in the field as it enables researchers to identify themselves within their own cultures. In addition, this step requires responses to two key questions: (a) How do researchers’ personal experiences impact the design, conduct, and findings of their MMR studies? (b) How does culture impact the conceptualization and implementation of MMR designs? After adequately answering these two questions, the researcher could be considered an agent of change capable of designing and directing culturally sensitive MMR studies. In this context, cultural responsiveness refers to being aware of the singularity and essence of the participants’ contexts and the internal variability of reality and external circumstances that influence cultural methodological frameworks.

Developing Culturally Responsive Training and Publications for Best Mixed Methods Research Practices One of the most significant challenges MMR researchers face from developing or non-Englishspeaking countries is the lack of MMR training or journals in their language. Therefore, promoting workshops led by local trainers or the creation of

Angel, R. J. (2013). After Babel: Language and the fundamental challenges of comparative aging research. Journal of Cross-Cultural Gerontology, 28(3), 223–238. https://doi.org/10.1007/s10823013-9197-2 Creswell, J. W., & Plano-Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Creswell, J. W., & Sinley, R. C. (2017). Developing a culturally-specific mixed methods approach to global research. Kolner Zeitschrift fur Soziologie und Sozialpsychologie, 69, 87–105. https://doi. org/10.1007/s11577-017-0453-2 Denscombe, M. (2008). Communities of practice: A research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2(3), 270–283. https://doi.org/10.1177/155868 9808316807 Escalante-Barrios, E. L., Suarez-Enciso, S. M., Raikes, H., Davis, D., Garcia, A., Gonen, M., Veziroglu-Celik, M., & Hazar, R. G. (2020). Child–parent interactions in American and Turkish families: Examining measurement invariance analysis of Child-parent Relationship Scale. PLoS ONE, 15(4), Article e0230831. https://doi.org/10.1371/journal.pone. 0230831 Escalante-Barrios, E. L., Suárez-Enciso, S. M., Putnam, S. P., Raikes, H., & Fàbregues, S. (2021). Using the Very Short Form of the Children’s Behavior Questionnaire for Spanish-Speaking Populations in Low- and Middle-Income Countries: A Psychometric

466

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Analysis of Dichotomized Variables. Children, 8(2), 1–14. http://doi.org/10.3390/children8020074 Fàbregues, S., Escalante-Barrios, E. L., Molina-Azorín, J. F., Hong, Q. N., & Verd, J. M. (2021). Taking a critical stance towards mixed methods research: A cross-disciplinary qualitative secondary analysis of researchers’ views. PLoS ONE, 16(7 July), Article e0252014. https://doi.org/10.1371/journal.pone. 0252014 Harris, J. (2022). Mixed methods research in developing country contexts: Lessons from field research

in six countries across Africa and the Caribbean. Journal of Mixed Methods Research, 16(2), 165– 182. https://doi.org/10.1177/15586898211032825 Helman, C. (1994). Culture, health, and illness: An introduction for health professionals (3rd ed.). Butterworth–Heinemann. Nastasi, B. K., & Hitchcock, J. H. (2016). Mixed methods research and culture-specific interventions: Program design and evaluation. Sage. Plano-Clark, V. L., & Ivankova, N. V. (2016). Mixed methods research: A guide to the field. Sage.

SECTION 6

Exploring Design Possibilities and Challenges for Mixed Methods Research

This page intentionally left blank

Exploring Possibilities and Challenges for Mixed Methods Research for the Future: Section 6 Introduction Peter Rawlins and Maggie Hartnett

This sixth and final section of the Handbook identifies and discusses the opportunities and challenges for MMR for the future. This section is about new things coming together in ways we have not yet seen or existing things coming together in new ways. In a rapidly changing world, where data is everywhere, and people and technology are interdependent in ways that we could not have imagined, how do we advance MMR designs to understand and solve complex and wicked problems? How can we build capacity within the global MMR community and teams to leverage opportunities and mitigate challenges facing them? In this section we explore the interconnection between three threads: evidence, people and technology. The evidence thread asks the question: What might constitute credible evidence in the future? The people thread explores capacity within community and teams. The technology thread examines the potential of existing data that are generated while living in the modern world. Each chapter concludes with imagining possibilities within the future of mixed methods research designs. This section has five chapters (see Table S6.1). The first four chapters cover the following topics: the use of visuals in the teaching and learning of mixed methods research, the use of mixed methods research to solve grand challenges for the betterment of society, the application of mixed methods

in translational research, the potential for and benefits of transdisciplinary mixed methods research. We had identified these as areas where we saw a lot of potential for future growth and development within the field of mixed methods research. The fifth chapter looks at the current state of mixed methods designs represented by this Handbook, and offers our projection of four emerging directions for mixed methods research design. In the first chapter, Peggy Shannon-Baker argues that the teaching and learning of mixed methods research can be improved through the use of visuals (e.g., diagrams of research designs, joint displays for integration). This chapter, linked to the evidence, technology and the people thread, draws on Shannon-Baker’s extensive experiences as a student, a teacher and a researcher of mixed methods to discuss how they use visuals as a pedagogical tool to help emerging researchers build their capacity to understand and undertake mixed methods research. In Chapter 32, Shannon-Baker outlines how learners come to the study of mixed methods research from diverse backgrounds, understandings and experiences of learning other research methodologies, often leading to fear of and misconceptions about research methodologies, and concerns about methodological incompatibility. Their argument is that the use of visuals can help to address misconceptions about mixed methods research and bolster

470

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table S6.1  Summary of Section 6 chapters: Exploring design possibilities and challenges for mixed methods research Chapter authors (country affiliation)

Chapter title

Peggy Shannon-Baker (USA)

Visualising the Process: Using Visuals to Teach and Learn Mixed Methods Research Toward the Future Legitimacy of Mixed Methods Designs: Responsible Mixed Methods Research for Tackling Grand Challenges for the Betterment of Society Realizing Methodological Potentials and Advantages of Mixed Methods Research Design for Knowledge Translation

José F. Molina-Azorin (Spain) and Michael D. Fetters (USA) Nataliya V. Ivankova, Jami L. Anderson, Ivan I. Herbey, Linda A. Roussel, and Daniel Kim (USA) Mandy Archibald (Canada) John Creswell (USA), Cheryl N. Poth (Canada), and Peter Rawlins (New Zealand)

students’ foundational knowledge of both the methods being mixed and mixed methods research as a methodology in its own right. They outline how visuals can be used to teach key features of mixed methods research such as timing, sequence, priority and integration. Prefacing this discussion, ShannonBaker examines the history of teaching mixed methods research, including concepts of course content, teacher pedagogy, assessment and the use of visuals in teaching mixed methods research. This discussion is illustrated with practical examples from postgraduate courses in research methods, including mixed methods research. Shannon-Baker concludes with a discussion of the challenges that teachers of mixed methods research, those undertaking mixed methods research, and the dissemination of mixed methods that research will face in the future. Readers will find this chapter useful for providing access to the potential of visualisations in mixed methods research. In the second chapter, José Molina-Azorin and Mikael Fetters examine how mixed methods research should be used responsibly to study grand societal challenges for the betterment of that society. Linked to the evidence and people threads, the authors argue that researchers must reconsider what constitutes impactful research, expanding the view from the traditional academic impact of research (e.g., publications and citations) to include practical and societal impacts through responsible and transformative research. In Chapter 33, MolinaAzorin and Fetters discuss a set of principles that can be used to highlight the practical and relevant aspects of social research. These principles include that research should serve society, value both basic and applied contributions, value plurality and multidisciplinary collaboration, use sound methodology, involve stakeholders, emphasise research with an impact on stakeholders, conduct ethically sound

Opportunities and Challenges for a Transdisciplinary Mixed Methods Research Future Mapping the Current State and Emerging Directions for Mixed Methods Research Design Using this Handbook

research, and broadly disseminate the findings of research to stakeholders. They then provide a set of examples of mixed methods studies that have sought to address grand challenges with these principles in mind. These include studies on poverty, environmental degradation, equity for marginalised populations, migration and the COVID-19 pandemic. Readers will find this chapter useful for making the “added value” benefit of mixed methods research. The third chapter, by Ivankova, Anderson, Herbey, Roussel, and Kim, investigates the potential of translational research, in the field of mixed methods research. Translational research, already well established in the health sciences area, focuses on moving knowledge from the basic sciences to its application in clinical and community settings. Linked to the people thread, this chapter seeks to explore how mixed methods research might facilitate the development and dissemination of this knowledge, and what the implications for mixed methods research designs might be. It draws on five translational practices—evidence-based practice, adaptation, dissemination and implementation, community-based participatory research and action research—to examine how mixed methods research can augment translational research through the nexus of epistemology, axiology, methodology and research designs. It is through this nexus, they argue, that integrative thinking for addressing complex problems can occur, thereby facilitating the process of knowledge translation. To achieve this, the authors discuss a mixed methods framework for translational research, illustrating their discussion with real-world examples and practices from translational science researchers who actively use mixed methods research approaches. They conclude the chapter by considering some additional aspects for designing mixed methods studies in translational

SECTION 6 INTRODUCTION

research. Readers will find this chapter useful for providing a rationale for both translational and mixed methods research. The fourth chapter, by Mandy Archibald, looks at the opportunities and challenges for a transdisciplinary mixed methods future. The chapter links to the people thread by examining the extent of integration required by researchers working at the intersection of transdisciplinary and mixed methods research. In Chapter 35, Archibald argues that, while mixed methods and transdisciplinary research continue to evolve as separate methodological orientations, they share a number of essential principles. As such when combined into transdisciplinary mixed methods research, they have the potential to address wicked social problems, and in doing so create impactful realworld research. Archibald introduces transdisciplinary mixed methods research as a collaborative approach emphasising the human dimensions of integration that underpin collaborative research. Drawing on concepts of integration, reflexive practice, innovative mergers, tailored facilitation, recursive designs and realist principles, she highlights key philosophical as well as practical considerations to guide transdisciplinary mixed methods research practices. Archibald also considers the use of technology for bringing together research teams and supporting team research more broadly. Readers will find this chapter useful for guiding their work as transdisciplinary mixed methods researchers. The final chapter, by Creswell, Poth, and Rawlins, is focused on the future directions for mixed methods research. The concept for this chapter arose out of the 2022 MMIRA global conference. During this conference, the current handbook was introduced and unpacked during a symposium session. Given his history in the field, Creswell was invited to be the discussant for this symposium. The final chapter is centered around his observations in that role. The chapter starts by unpacking the concept of “mapping” as a tool to examine the current state of a given field of research. This is followed by a section which looks at how past mixed methods research handbooks and conferences have been used as “current state” indicators of the field of mixed methods research. Chapter 36, Creswell et al. identify the key characteristics and unique features of past handbooks and argue that these provide us with insight into the main areas of interest of mixed methods research at the time of those handbooks and conferences. This section ends with a summary of the current Handbook, the key questions and unique issues raised in each of its sections. Next, the chapter revisits the 2016 MMIRA task force report (Mertens et  al., 2016) commissioned by Creswell as the then president of MMIRA. The task force

471

report identified four key areas the mixed method community might usefully engage with over the subsequent five years: conceptual and methodological advances; technology and big data; preparation of mixed methods researchers; and complex social problems. Keeping with the mapping theme, the chapter then maps the symposium discussant themes against the four areas in the MMIRA task force report and gives examples of topics covered and examples of chapters in the Handbook. This leads to a final section where Creswell, Poth and Rawlins identify and map four emerging directions for mixed methods research design afforded by the Handbook: embracing emergent, flexible and uncertain designs; valuing international applications and cultural adaptations in designs; describing innovative technology applications in designs; and addressing societal issues with new design intersections and practices. Readers will find this chapter useful for thinking about the future directions their own research might take. Collectively, the chapters in this section seek some new direction in the future of mixed methods research over the next 5–15 years. In reality, this is something that proved more difficult than we had imagined. Recent changes in the way that people interact and work with each other brought about by the COVID-19 pandemic have, in our view, created a degree of uncertainty about future directions. What is clear in the chapters that follow is that the role of evidence, people and technology will be central to this future. As social media, big data and Artificial intelligence (AI) become ubiquitous, we can expect to see new sources of data and new ways of analysing this data. A focus on involving communities and other stakeholders in addressing society’s grand challenges will see the development of larger research teams and networks. Additionally, technology will play a role in both the collection and facilitation of evidence and the collation and operation of research teams/ networks. We will revisit these key themes in our concluding thoughts.

REFERENCE Mertens, D. M., Bazeley, P., Bowleg, L., Fielding, N. G., Maxwell, J. A., Molina-Azor, J. F., & Niglas, K. (2016). Expanding thinking through a kaleidoscopic look into the future: Implications of the Mixed Methods International Research Association’s task force report on the future of mixed methods research. Journal of Mixed Methods Research, 10(3), 221–227. https://doi.org/ 10.1177/1558689816649719

32 Visualizing the Process: Using Visuals to Teach and Learn Mixed Methods Research Peggy Shannon-Baker

INTRODUCTION Improving teaching about mixed methods has a direct connection to future uses of and innovations in this methodology. Using creative approaches to teaching mixed methods, including the use of visuals in the teaching and learning process, can build emerging researchers’ capacities to imagine innovative mixed method practices and designs to address complex problems. Therefore, considerations of the future possibilities and challenges for mixed methods theory and practice need to also attend to those faced by mixed methods instructors who directly shape the field’s future. The purpose of this chapter is to argue for the value of using visuals to teach and learn about mixed methods research. For the purposes of this chapter, I define visuals as figures, tables, or other line drawings that use shapes, arrows, lines, and other elements to communicate information. In this chapter, I provide an overview of common practices for teaching research methods and mixed methods in particular. However, since mixed methods is an evolving field, this discussion is based on practices and conceptualizations for mixed methods research common at the time of writing this chapter. I also provide a discussion of the value of visuals in this process, and how I have

used visuals to teach specific mixed methods elements including design-specific nuances related to timing and integration approaches. Investigating how research methods are taught is important because these courses communicate a set of values and the positioning of various methods in the field and programme (Capraro & Thompson, 2008; Tashakkori & Teddlie, 2003). Additionally, these norms also impact the peer review process and the “quality and usefulness of published articles” (Capraro & Thompson, 2008, p. 247). Learning environments offer spaces for emerging researchers to experiment with ideas (Niglas, 2009) and improve their skills in the communication, design, and implementation of mixed methods. Reviewing the current and future implications for technology in visualization is beyond the scope of this chapter. Instead, this chapter focuses on capacity building in people by discussing how teachers of mixed methods research can operationalize the benefits of making visuals in their teaching. I conclude with some of the key topics and challenges mixed methods researchers and teachers will face in the use of visuals and visualizations in the future. This chapter is based on my own experiences as a student, teacher, and researcher of mixed methods. I come to this work with training across several research methodologies (mixed methods, arts-based, qualitative, and quantitative research),

VISUALIZING THE PROCESS

experience as a student and teacher in multiple formats (completely online, hybrid or blended, and face-to-face), and experience teaching research methods in educational programmes open to students from other fields (e.g., psychology, public health). I currently teach completely online or hybrid courses at the Master’s level and advanced graduate or doctoral courses specifically on mixed methods, qualitative, and action research. I also mentor students’ thesis and doctoral research projects that have used methods across my four methodological background areas. Aside from my own teaching and learning experiences, this chapter also builds from the three decades long discussion on the teaching of qualitative methods (cf. Webb & Glesne, 1992) by turning this line of inquiry to the teaching of mixed methods.

FOUNDATIONS IN THE TEACHING OF RESEARCH DESIGNS AND METHODS IN SOCIAL AND BEHAVIOURAL SCIENCES Some have argued that introductory research methods courses should be framed from a mixed methods perspective in general (Tashakkori & Teddlie, 2003), that qualitative and quantitative research can be presented on a continuum to equip learners with research skills for any scenario (Niglas, 2007), or that “qualitative” and “quantitative” should not be frames to teach research methods at all (Onwuegbuzie & Leech, 2005). Learners refers to individuals learning mixed methods research in a variety of contexts including in university courses, workshops, professional development seminars, etc. Despite these wide-ranging views of how to frame courses on research design and methods, there are some commonalities among these courses. At the Master’s and doctoral level, these courses focus on building learners’ methodological competence including literacy, reasoning, and thinking within the methodological norms for their field (delMas, 2002; Malin et al., 2007). Courses emphasizing methodological competence centre around developing learners’ abilities to critically read research, distinguish between methodologies and designs, and design a future study, depending on time available for the course (Murtonen, 2015). Instructors of courses with a methodological competence focus use course activities and assessments that measure actionbased rather than content-based expectations (delMas, 2002), such as giving a research project presentation, writing a paper intended for publication, and writing a grant proposal (Capraro & Thompson, 2008). More advanced research methods courses might focus on mapping the historical

473

development and key debates in the methodological field (e.g., Plano Clark & Ivankova, 2016). Additionally, courses on research design and methods are taught in a variety of formats, including online, face-to-face, and a hybrid of the two, may be organized in a two-part sequence to cover more advanced topics and/or skills in the second course, and can even include the use of games and simulations in learning about research methods (Boyle et al., 2014).

TEACHING AND LEARNING MIXED METHODS RESEARCH Whereas one course might cover multiple methods (e.g., both qualitative and quantitative approaches in a single course), a course on mixed methods research focuses on the integration of multiple methods (Bazeley, 2003) and the unique features of this methodology. The first known course focused solely on mixed methods research is said to have been taught at Valdosta State University (Valdosta, Georgia, USA) by Richard Schmertzing and Marsha Reed in 2000 (Creswell et  al., 2003). Mixed methods standalone courses may be offered at the master’s and doctoral levels (Bazeley, 2003; Creswell et al., 2003; Earley, 2007; Onwuegbuzie & Leech, 2005; Tashakkori & Teddlie, 2003), however, this is not the case for many institutions worldwide. A lack of such courses might be the result of many factors, including the relative youth of the field of mixed methods research, limited faculty expertise to teach such courses, limited space available in programmes for an additional course, etc. Many of the challenges that instructors face in teaching research courses are related to those faced by teaching mixed methods research specifically. However, mixed methods instructors face additional challenges due to the unique features of this research methodology. To ask what is unique to the teaching and learning of mixed methods research, one must also ask what is unique to mixed methods research compared to other methodologies. This includes needing a foundation in the research methods being mixed, understanding the field of mixed methods research, and identifying meaningful ways to integrate in a mixed methods study.

From a Foundation in What is Being Mixed Since mixed methods research involves the mixing of multiple methodological approaches, many mixed methods courses have a prerequisite that

474

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

learners come with a foundational understanding of the approaches to be mixed (Creswell et  al., 2003; Niglas, 2009). This prior experience can either be in the form of a course or research experience (Christ, 2009). With this foundation, teachers can then provide a review of the methodologies at the beginning of the term to supplement learners’ existing knowledge and skills, but then otherwise focus the rest of the course specifically on mixed methods research (Earley, 2007). This foundation in the methods being mixed includes the characteristics, uses, benefits, challenges, and forms of rigour specific to those approaches. Learners need to be able to critically read, design, implement, and evaluate these methods in their own and others’ work. These skills are the foundation that mixed methods courses often build upon.

Mapping the Field and Unique Elements of Mixed Methods Research Courses about mixed methods explore the conceptualizations of this methodology in the field and its unique elements. This includes how to define and rationalize the use of mixed methods, how to formulate mixed methods research questions, selection of a specific mixed methods design, mixed methods sampling strategies, types of integration, influence of methodological priority, methodological timing, and specific forms of mixed methods validity and rigour (Bazeley, 2003; Ivankova, 2010; Poth, 2014; Tashakkori & Teddlie, 2003). More advanced courses at the doctoral level may also review key debates in the field, historical developments, current innovations, and the influence of philosophical paradigms on the mixed methods research process (Bazeley, 2003; Creswell et  al., 2003; Earley, 2007; Ivankova, 2010; Ivankova & Plano Clark, 2018; Plano Clark & Ivankova, 2016; Poth, 2014). Integration is a key component of mixed methods research and courses on this methodology. For the purposes of this chapter, I define integration as the intentional interface between two or more methodological approaches in a mixed methods study, though integration can occur at multiple levels (Creswell et al., 2003; Fetters et al., 2013). Learners of mixed methods research need examples of and opportunities to practise the different types of integration (Creswell et  al., 2003). Teaching about integration entails developing learners’ ability to identify appropriate integration strategies that fit the research questions, methodologies used, types of data collected, and how the data were originally analyzed. Thus, teaching about integration requires a strong foundation in

the methods being mixed to identify common and new methods of integration.

Pedagogy and Assessment in Mixed Methods Research Courses Pedagogy refers to one’s teaching practices, including teaching strategies, curriculum content design, assessment strategies, and the teacher’s overall role in the classroom. The pedagogy of mixed methods research is informed by how instructors define mixed methods research (Bazeley, 2003), including when and what is mixed and the relative importance of other features of this methodology such as using a typology-based mixed methods design, integration, timing, working in teams, and priority. Although much of the research on teaching mixed methods research focuses on student learning and practical outcomes, some identify instructors’ teaching strategies. Common teaching strategies include lecture (Christ, 2009; Creswell et al., 2003; Earley, 2007; Poth, 2014) and in-person or online discussions (Creswell et al., 2003; Ivankova, 2010; Poth, 2014) as well as guest presentations from researchers on published or ongoing mixed methods studies (Creswell et  al., 2003). Others have used a team-teaching approach where two faculty coteach the course to mitigate the tendency for a course to be taught from a faculty member with primarily one-sided methodological training (Onwuegbuzie & Leech, 2005). Informal and formal assessment strategies vary widely among instructors of mixed methods. Some scholars recommend using general assignments such as class discussions, reflective writing, and written responses to readings (e.g., Earley, 2007; Ivankova, 2010; Ivankova & Plano Clark, 2018). Other common assessments include writing a methodological or topic-based paper with the goal of later publishing it (Creswell et al., 2003; Ivankova & Plano Clark, 2018; Poth, 2014), evaluating published studies (Creswell et  al., 2003; Earley, 2007; Ivankova, 2010; Niglas, 2007; Poth, 2014), collecting data or working with sample data to practise integration and interpretation strategies (Bazeley, 2003; Creswell et  al., 2003), creating a study proposal (Christ, 2009; Creswell et  al., 2003; Ivankova, 2010; Poth, 2014), and conducting small-scale studies (Bazeley, 2003; Niglas, 2007). These approaches mirror assessment strategies used to teach courses on any research methodology. Other activities more specific to mixed methods research include discussing where to publish mixed methods empirical studies (Bazeley, 2003; Poth, 2014); given the length needed

VISUALIZING THE PROCESS

to appropriately describe the methods and account for the breadth of findings as well as the relative youth of the methodology and its reception in some fields, mixed methods researchers face challenges publishing their work outside of methodological journals. Additionally, learners of mixed methods research also evaluate single methodology designs to see how adding another strand or phase might have impacted the original study (Creswell et al., 2003) and work in small groups to develop their skills in working in diverse research teams (Bazeley, 2003).

Challenges Instructors of Mixed Methods Research Face Learners come to mixed methods courses with varying backgrounds, understandings, and experiences of other research methodologies. Mixed methods learners should be familiar with multiple methodological traditions but not in a prejudicial way toward a methodology (Bazeley, 2003). Like courses about other research methodologies, learners may come to mixed methods courses with fears and learned misconceptions about specific research methods (Leech et al., 2007) or that the rigour or quality of certain methodologies is inherently limited. Additionally, teachers of mixed methods also have to navigate tensions with learners who have learned research methods from an incompatibility perspective—i.e., that research methods and the research philosophies informing them are so diametrically opposed that they cannot be combined in a single study (Tashakkori & Teddlie, 2003). Learners may also have faced discouragement from mentors in pursuing mixed methods due to concerns about time needed to complete a mixed methods study, the feasibility of using a mixed methods design, or the validity of this methodology. As a result, mixed methods teachers must simultaneously rationalize and support the individual methodologies they teach as being mixable in addition to teaching the validity to using mixed methods research.

475

knowledge and understanding of the related concepts and the overall structure of the visual are important in the ability to interpret a visual (Cook, 2006; Shah & Hoeffner, 2002). This literature focuses on graphic representations of statistical data or analyses (Koerber, 2007) as well as the pedagogical side to using visuals. For example, visual representations help learners understand concepts in new ways (cf. Cook, 2006). Learners’ level of understanding can be assessed by having them create visuals. For example, Murtonen (2015) compared novice, advanced, and expert knowledge levels about statistics and research methodology by asking participants to define research and draw a concept map for its meaning. Murtonen found that the maps drawn by experts displayed more connections between components, as well as the impact of informal knowledge on the research process. In this study, the visual method of the concept map helped assess the different level of learners’ understanding. Although the literature on teaching and learning about mixed methods research is sparse, much of this literature includes examples of how visuals have been used in learning opportunities. For example, instructors present typical research diagram formats when they teach about mixed methods designs (Poth, 2014). Instructors also use visuals to assess learners’ understanding of design types. Instructors of mixed methods research have learners create research diagrams for proposed or published research studies to communicate their understandings about the timing and integration in the study (Christ, 2009; Creswell et al., 2003; Ivankova, 2010). These research diagrams can be created using a pre-existing format or graphic organizer (cf. Christ, 2009; Fetters, 2020) or by having learners draw their diagrams from scratch in a computer program or by hand. Additionally, courses that use visuals can help learners develop their skills in analyzing visual data through the use of visually oriented frames of analysis (e.g., Pauwels, 2010).

WHEN AND HOW I USE VISUALS TO TEACH MIXED METHODS RESEARCH Use of Visuals in Teaching Research Methods Researchers in the area of visual literacy in learning about research design and methods identify the importance of learners’ ability to both interpret visuals or graphic representations and identify when to use specific types of representations (Friel et al., 2001; Shah & Hoeffner, 2002). Prior

For this section, I focus on how I have assigned visuals as part of a mixed level graduate course on mixed methods research. This course was a fully online and asynchronous course I taught in Spring 2021 in the Desire2Learn Learning Management System hosted by my university. The course had a mix of learners at the Master’s, Educational Specialist, and doctoral levels. Learners were

476

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

enrolled in numerous programmes, including evaluation and assessment, educational leadership, elementary education, and psychology. The course filled an elective requirement or research methods requirement for these degree programmes. I taught the course using filmed video lectures, walk-through videos for how to read empirical mixed methods studies, publicly available videos from other mixed methods scholars, and small- and large-group discussions. I used assessments that started with learners’ own definitions for mixed methods research, identifying and critiquing published use of mixed methods designs, and practising mixed methods skills in integration and designing mixed methods studies. I aimed for the course to embody a mix of learning the conceptual components of mixed methods research (e.g., timing, common design types) with a strong practical and applied focus, given that many learners came into the class with an interest in using mixed methods approaches in their professional work. I share students’ visuals using their self-chosen pseudonyms from an Institutional Review Board-approved study I conducted on the course.

Teaching with Visuals Examples: Research Diagrams Creating research diagrams that communicate the timing and sequence of steps in a study is a key skill for mixed methods researchers (Christ, 2009; Creswell et  al., 2003; Ivankova, 2010). In this mixed methods course, I asked learners to make research diagrams for two types of assignments. The first type of assignment was a “Design Critique” where learners locate empirical articles on a topic of their choice that use a specific mixed methods design. This assignment followed lectures from me, guest videos, and readings introducing common mixed methods design types, their distinctions, reasons for using each design type, and how researchers write about and notate these designs. In the Design Critique, learners annotated the article they chose, identifying key information about the methods used, definition and rationale for mixed methods, and impact of using this approach. They completed two Design Critiques: one on a convergent design (where two or more methodological approaches are used and analyzed separately until integration; Creswell & Plano Clark, 2017) and one on any kind of sequential design (where one methodological phase is used to directly influence the next phase; Creswell & Plano Clark, 2017). They then wrote a paper that: described what they knew about the design type,

cited the relevant mixed methods literature, summarized the methods for their chosen study, and critiqued their chosen example’s use of mixed methods in general and in relation to Onwuegbuzie and Johnson’s (2006) framework for forms of mixed methods legitimation. Learners also created a research diagram for their Design Critiques based on the information presented in the article. Since this as possibly the first time that students have made a research diagram or seen one in a published study, I reviewed the basic design features of a diagram such as arrows to indicate a sequence, what types of information are included in basic versus detailed diagrams (e.g., listing general methods vs. also listing outcomes or products for a research phase), and where mixed methods notation might be listed in the diagram. I showcased some examples of basic and detailed research diagrams (e.g., Liao & Li, 2020; Maleku et al., 2021). For example, Jannet created a research diagram based on a convergent mixed methods study by Patel et al. (2016) on the interactions between family stressors and academic outcomes for adolescent recent immigrants (see Figure 32.1). When assessing their research diagrams in the Design Critiques, I initially looked for accuracy of research methods, design type, priority, and notation. This required reviewing their chosen article and annotations against the text of their critique and diagram details. I provided more detailed feedback on the diagrams earlier in the term to encourage greater detail and accuracy in their text and visual elements, such as the accurate use of mixed methods notation and clearly labelling data collection, analysis and integration procedures. In Jannet’s example, she used the research diagram to communicate the findings from the analyses related to each research strand and the integration of findings from the study by Patel et al. (2016). The final “interpretation” section included a reflection on the implication of the methods used in identifying the particular impact of family separation for new immigrant youth. Students in this course also created a detailed “Mock Mixed Methods Research Design Proposal” (cf. Creswell et  al., 2003; Ivankova, 2010; Poth, 2014). This proposal as a culminating project designed to have them demonstrate their cumulative knowledge about and skill in designing a mixed methods study. They chose a topic of interest to them personally and/or professionally, shared a draft of their proposal for feedback from peers in the class, and posted their revised proposal as the course final. The proposal components included: a background that briefly introduced the topic; research purpose and questions; a research methods section that included an introduction to how

VISUALIZING THE PROCESS

477

Figure 32.1  Jannet’s research diagram based on Patel et al. (2016) Source: Used with permission from research participant.

they defined mixed methods and their reason for using it in this study, the specific mixed methods design they would use, a detailed research diagram, relevant details for data collection and analysis, and their mixed methods integration procedure(s); and a reflection on designing the proposed study. Learners’ research diagrams for this final tended to be more detailed than the previous research diagrams they created in the Design Critique papers, possibly due to their increased familiarity with using this visual method of communication and/or their enhanced understanding of mixed methods designs (cf. Murtonen, 2015). In their reflection on the proposed study, I asked them to discuss what excites them about their study, what challenges they foresee implementing it in the future, and any other reflections they want to share. Brandi Williams created a detailed diagram for her proposed explanatory sequential (QUANàQUAL) study about learners’ and teachers’ perspectives on how learner performance was impacted by the COVID-19 pandemic (see Figure 32.2). Her diagram showcases several key elements for mixed methods studies: identifying a purpose for each phase, specifying clearly how one phase influences the next in a sequential design, and using multiple forms of integration. Reflecting on creating this diagram, she shared, “I felt like I tapped into my skills as a prospective

researcher. It was exciting to have a heightened sense of confidence for conducting a mixed methods study.”

TEACHING WITH VISUALS EXAMPLES: JOINT DISPLAYS FOR INTEGRATION Integration is another key component for emerging mixed methods researchers to learn and practise (Creswell et al., 2003). Beyond defining and identifying different types of integration, learners in this course practise three specific forms of integration: merging integration (which refers to comparatively analyzing two sets of findings especially via creating a joint display), building integration (which, in this case, refers to using the findings from a previous research phase to inform the data collection of the next research phase), and transforming integration (which refers to converting one form of data or findings into another form of data for further analysis) (Creswell & Plano Clark, 2017). In an assignment for merging integration, they chose from a list of published mixed methods studies I provided that do not have a merging joint display or a visual that showcases how the data compared and contrasted with each other across research strands (Plano

478

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 32.2  Brandi’s research diagram included in her mock mixed methods research design proposal final Source: Used with permission from research participant. Note: In the original version of the image for Figure 32.2, Brandi used blue boxes for the quantitative Phase 1, yellow for the qualitative Phase 2 boxes and green for the final integration box.

Clark & Sanders, 2015). I reviewed mixed methods articles published within the last five years of the course related to learners’ programme areas and research interests shared at the beginning of the term. I aimed to identify articles with a clear discussion of their findings and how they compared or contrasted across research strands. These articles often have integrated findings, meaning that they refer to both qualitative and quantitative evidence for a meta-theme in their findings section (e.g., Casanova et al., 2018). They then chose one article to read deeply, interpret the findings, identify convergence and divergence in the findings and communicate these convergences/divergences in a merging joint display. Karis, for example, chose the article by Olkin et  al. (2019) about the microaggressions women with disabilities face (see Figure 32.3). Her joint display used a colour-coded compound graphic modelled after Haynes-Brown and Fetters (2021) to compare quantitative survey results to qualitative focus group themes. Reflecting on the assignment, Karis shared that “the effort to identify and connect the quantitative and qualitative data deepened my understanding of the impact of microaggressions on persons with disabilities.” This joint display showcases how some qualitative and quantitative findings from Olkin et al. (2019) could be connected whereas others were not as

clear to Karis. Making a joint display based on another researcher’s study cemented the importance of having clear and detailed findings and the overall challenge of merging qualitative and quantitative findings together. Next, learners practised building integration. In this assignment, they designed the second phase of a study based on a pre-existing study that used a single methodology. To communicate how the second phase (imagined by the learner) built from the findings of the first phase (i.e., the published article), they created a building joint display. By designing this second study phase and the building joint display, learners expanded upon their existing skills in critically reading empirical studies, making intentional decisions about mixed methods designs, and visually communicating their ideas in a visual joint display. These are skills they began developing through the Design Critiques and previous integration assignment. For the building integration assignment, learners chose an article on their topic of interest that uses a single methodology (e.g., quantitative research alone). Likely their topics have published articles that use a number of different methodologies, so should they chose one that starts from qualitative research, quantitative research

VISUALIZING THE PROCESS

479

Figure 32.3  Karis’s merging integration joint display based on Olkin et al. (2019) Source: Used with permission from research participant.

or something else? I told them to make an intentional choice in selecting their article since they will need to design a second phase building on that study in some other methodology. They could choose an article that ends up having them design a second phase in a methodology they wanted to practise more, or they could choose an article where the second phase they designed was more in their comfort zone methodologically. In other words, learners chose their own level of challenge and what type of research methodology to use for the second phase of their imagined study. Then, for the second phase, they outlined: a research purpose and questions or hypothesis, sampling procedures, data collection procedures and analyses. Learners then posted their building joint display, a summary of the methods from their chosen article, and their second phase methods in an open format post in our online discussion board. Sasha, for example, designed an exploratory sequential mixed methods study based on a qualitative phenomenological study by RobinsonWood et  al. (2015) about the microaggressions highly educated Black women face. RobinsonWood et al. (2015) identified that further research is needed into hair-based racial and gender microaggressions Black women face, which Sasha used as the driving factor for her design of a followup quantitative phase from their study. Sasha then used her own literature review into hair and personal appearance based microaggressions to identify five domains for which she created mock

survey items (see Figure 32.4). In her reflection on the assignment, Sasha shared that she chose to design the quantitative phase for a study to further enhance her quantitative skills, since this methodology “does not come naturally” to her. She also shared that “thinking back on the procedure diagrams we created earlier in the class [helped her] be reminded of the purpose each [research phase] serves” in the larger mixed methods study. In other words, creating other visuals throughout the semester provided her with examples to work from and reinforced the importance of identifying a clear research purpose for each methodological strand in a mixed methods study.

ENVISIONING FUTURE ISSUES IN THE USE OF VISUALS IN (TEACHING) MIXED METHODS RESEARCH Envisioning future uses of visuals means asking how visuals will become more integrated into the field and use of mixed methods research (cf. Shannon-Baker, 2022) as well as how to use more visuals in teaching. In this section, I identify key issues I foresee playing a major role in the use of visuals in mixed methods and thereby the teaching of mixed methods research: using social media generated visual data; asking key questions about data sovereignty, ethics and

480

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 32.4  Sasha’s building integration joint display for adding a quantitative phase to Robinson-Wood et al. (2015) Source: Used with permission from research participant.

informed consent in the use and publication of visuals; using software analysis programs critically; expanding publication limits on length and visuals; and who and what to watch in the future for more developments around visualizations and teaching mixed methods research.

Use and Analysis of Visual Data In the next 5–10 years, mixed methods researchers will use more social media data as well as the use of visual (and other) data created by artificial intelligence (AI). Social media is becoming increasingly visually oriented with emojis, memes, gifs, videos, and photographs. I anticipate seeing more social media-generated data, especially from younger researchers, as well as research involving younger populations. The increased use of social media visual data will necessitate discussions about data sovereignty (Carroll et al., 2019), engaging in ongoing and informed consent practices with the people who created the data, who can access and use such

data, and how to display others’ data. At the time of publishing this chapter, there has been a tremendous increase in the number of AI tools generating information for users. Soon this information will include data sets (including visual data) that mixed methods researchers may use. How will such AI-based data be generated and used? This means that mixed methods researchers will not only need to be able to create visuals to design and share their mixed methods studies, but also be trained on how to use and interpret existing visuals. Within 15 years, we could see examples of existing visual data in mixed methods studies in the form of augmented reality simulations from social media platforms and video games on gaming platforms or online. Instructors of mixed methods will need to have their own experience and training to facilitate these discussions with emerging mixed methods researchers. How will mixed methods research designs and research diagrams communicating the sequence of procedures adjust to these advances? Emerging mixed methods researchers will need to have capacity to collect, analyze, and interpret this data as well as find ways to communicate it with a broader

VISUALIZING THE PROCESS

audience. We will likely see developments in data analysis programs that will allow researchers to use algorithms or AI to analyze visual data descriptively (e.g., identify a piece of an image as a streetlamp) or analytically (e.g., identify the emotion on a face) (cf. Chapter 11, this volume). Some popular forms of AI at the time of writing this chapter like Lensa AI merge together visual data from online in ways that researchers can otherwise combine by hand (e.g., Shannon-Baker, 2021). Given concerns about how search engine algorithms perpetuate racist stereotypes (Noble, 2018), mixed methods researchers will need to engage in critical reflection in the use of these analysis programs. This includes asking within what cultural contexts these programs were developed since contexts influence the creation of algorithms and AI (Noble, 2018). As a result, teachers of mixed methods research will need to teach not only general analysis software skills, but also how to choose and use this software in a critical, ethical, and culturally relevant manner.

Data Visualization and Representation Mixed methods researchers will additionally need to consider how to create data visualizations that account for data that is increasingly robust and complex, whether due to the size of the data set or the use of multimodal data sets (Chapter 22, this volume). Although the course I highlighted in this chapter focused more on visualization in the design processes for mixed methods, future courses will need to consider how to create more complex, engaging, and appropriate data visualizations. In five years, we may see more researchers creating visuals such as infographics to communicate their study findings to a wider audience. Infographics are visual ways to represent information, study findings, or other data. These visuals rely heavily on graphics (e.g., flow charts, arrows, diagrams) to communicate information. Infographics can then be more readily shared on social media platforms, conferences, workshops, and websites to promote interaction with and use of a study’s findings. The issue of data visualization and representation will especially impact journal and book publishers. Journal reviewers and editors will need to be equipped to ask critical questions about the use of visuals in a potential publication, including how the visuals were created, how the writer(s) received appropriate consent to publish the visuals, and other methodological and ethical considerations to the use of visuals (cf. Shannon-Baker & Edwards, 2018). Publishers will need to expand their current policies around word/page limits and rules about

481

use of colour in published images. Length restrictions for journal articles already pose a challenge for mixed methods researchers. I suspect that as mixed methods research becomes more widespread and reviewers and editors alike are more knowledgeable about mixed methods, these limits may start to expand to allow for descriptions of more complex study methods. Specific to the use of visual data and visualizations, however, to adequately display and describe these visuals in published mixed methods research, researchers need additional space and the ability to publish visuals in full colour. Without full colour, researchers are limited in what can be communicated through a visual to readers. Publishers will need to offer space online or printing options to allow for colorful, and therefore more complex, visuals to be included in published work. Addressing these challenges to publishing mixed methods research that uses visual data will then impact the example studies that mixed methods instructors can showcase. Teaching with more complex, nuanced, and detailed published studies then has the potential to further expand the field because such examples would be shared with the future community of the field: learners of mixed methods research.

CONCLUSION The purpose of this chapter was to showcase how teaching mixed methods research with visuals can enhance the knowledge and skill capacity of emerging researchers. This discussion was based in the literature on how mixed methods research has been taught, including the content such courses cover, pedagogical strategies teachers use, the types of assessments used, and the unique chal­­ lenges instructors of mixed methods research face. These challenges include teaching learners from a wide range of methodological backgrounds, supplementing learners’ knowledge about the methods being mixed, and addressing misconceptions about mixed methods and the incompatibility of methods. I then showcased example visuals that learners created in a recent mixed methods research course I taught. These visuals included research diagrams and joint displays based on learners practising merging and building integration. Beyond showcasing visuals in my own pedagogy, this course demonstrated Bazeley’s (2003) call for mixed methods courses to include “experiential context as much as possible” (p. 121); learners read and designed studies in their own interest areas throughout the term. In the final section of this chapter, I offered some discussion of

482

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

the topics and challenges that mixed methods researchers and teachers of this methodology will face in the short- and long-term future. As visuals continue to play an important role in our everyday lives worldwide, incorporating them into research in meaningful ways will help research be more relevant. The use, analysis, and interpretation of visuals in mixed methods will then require courses to address critically creating and interpreting visuals, critical visual literacy, and the ethical and culturally relevant considerations of creating, using and publishing visuals.

WHAT TO READ NEXT Edwards, C. & Creamer, E. (Eds.) (2021). Innovative approaches to visual methods in mixed method research in psychological fields [Special issue]. Methods in Psychology, 5. https://www.sciencedire c t . c o m / j o u r n a l / m e t h o d s - i n - p s y c h o l o g y / special-issue/10SNRPCCG6Z

This special issue explores recent applications of and methodological considerations for employing visual methods in mixed methods research. Mixed Methods International Research Association Massive Open Online Course (MMIRA MOOC) in mixed methods research. https://mmira.wildapricot.org/

Since 2019, the MMIRA MOOC has offered free online, asynchronous modules in mixed methods research to members of the association. These modules are accessible to mixed methods enthusiasts at the foundational, intermediate, and expert levels of knowledge and skill. Frias, K. M., & Popovich, D. (2020). An experiential approach to teaching mixed methods research. Journal of Education for Business, 95(3), 193–205. https://doi.org/10.1080/08832323.2019.1627995

Aside from the articles cited in this chapter about teaching mixed methods research, this more recent work evaluates undergraduate courses that used experiential learning to teach students about mixed methods.

REFERENCES Bazeley, P. (2003). Teaching mixed methods. Qualitative Research Journal, 3, 117–126. https://search. informit.org/doi/abs/10.3316/aeipt.150146 Boyle, E. A., MacArthur, E. W., Connolly, T. M., Hainey, T., Manea, M., Kärki, A., & van Rosmalen,

P. (2014). A narrative literature review of games, animations and simulations to teach research methods and statistics. Computers & Education, 74, 1–14. https://doi.org/10.1016/j.compedu.2014. 01.004 Capraro, R. M., & Thompson, B. (2008). The educational researcher defined: What will future researchers be trained to do? The Journal of Educational Research, 101(4), 247–253. https://doi. org/10.3200/JOER.101.4.247-253 Carroll, S. R., Rodriguez-Lonebear, D., & Martinez, A. (2019). Indigenous data governance: Strategies from United States Native Nations. Data Science Journal, 18(1), 1–15. https://doi.org/10.5334/dsj2019-031 Casanova, S., McGuire, K. M., & Martin, M. (2018). “Why you throwing subs?”: An exploration of community college students’ immediate responses to microaggressions. Teachers College Record, 120(9), 1–48. https://search.ebscohost.com/login. aspx?direct=true&AuthType=ip,shib&db=eric& AN=EJ1163283&custid=gso1 Christ, T. W. (2009). Designing, teaching, and evaluating two complementary mixed methods research courses. Journal of Mixed Methods Research, 3(4), 292–325. https://doi.org/10.1177/155868980 9341796 Cook, M. P. (2006). Visual representations in science education: The influence of prior knowledge and cognitive load theory on instructional design principles. Science Education, 90(6), 1073–1091. https://doi.org/10.1002/sce.20164 Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). Sage. Creswell, J. W., Tashakkori, A., Jensen, K. D., & Shapley, K. L. (2003). Teaching mixed methods research: Practices, dilemmas, and challenges. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 619–637). Sage. delMas, R. C. (2002). Statistical literacy, reasoning, and thinking: A commentary. Journal of Statistics Education, 10(2), 13. https://doi.org/10.1080/ 10691898.2002.11910674 Earley, M. A. (2007). Developing a syllabus for a mixed-methods research course. International Journal of Social Research Methodology, 10(2), 145–162. https://doi.org/10.1080/136455707 01334118 Fetters, M. D. (2020). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Sage. Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designsPrinciples and practices. Health Services Research, 48(6pt2), 2134–2156. https://doi.org/10.1111/147 5-6773.12117

VISUALIZING THE PROCESS

Friel, S. N., Curcio, F. R., & Bright, G. W. (2001). Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education, 32(2), 124–158. https://doi.org/10.2307/749671 Haynes-Brown, T. K., & Fetters, M. D. (2021). Using joint display as an analytic process: An illustration using bar graphs joint displays from a mixed methods study of how beliefs shape secondary school teachers’ use of technology. International Journal of Qualitative Methods, 20, 1–14. https://doi.org/ 10.1177/1609406921993286 Ivankova, N. V. (2010). Teaching and learning mixed methods research in computer-mediated environment: Educational gains and challenges. International Journal of Multiple Research Approaches, 4(1), 49–65. https://doi.org/10.5172/mra.2010. 4.1.049 Ivankova, N. V., & Plano Clark, V. L. (2018). Teaching mixed methods research: Using a socio-ecological framework as a pedagogical approach for addressing the complexity of the field. International Journal of Social Research Methodology, 21(4), 409–424. https://doi.org/10.1080/13645579.2018. 1427604 Koerber, S. (2007). Student teachers’ ability to interpret and construct visual-graphic representations. In M. Murtonen, J. Rautopuro, & P. Väisänen (Eds.), Learning and teaching of research methods at university (pp. 51–69). Finnish Educational Research Association. Leech, N. L., Onwuegbuzie, A. J., Murtonen, M., Mikkilä-Erdmann, M., & Tähtinen, J. (2007). Researcher workshop for student teachers—An example of a mixed methods learning environment. In M. Murtonen, J. Rautopuro, & P. Väisänen (Eds.), Learning and teaching of research methods at university (pp. 205–226). Finnish Educational Research Association. Liao, H., & Li, Y. (2020). Intercultural teaching approaches and practices of Chinese teachers in English education: An exploratory mixed methods study. Language Teaching Research, Online First, 1–32. https://doi.org/10.1177/1362168820971467 Maleku, A., Kim, Y. K., Kagotho, N., & Lim, Y. (2021). Expanding the transformative explanatory sequential mixed methods design archetype in a crosscultural context: The Polemics of African refugee livelihoods in places of resettlement. Journal of Mixed Methods Research, 15(2), 212–239. https:// doi.org/10.1177/1558689820936378 Malin, A., Rautopuro, J., & Väisänen, P. (2007). Towards methodological competence in quantitative research methods education. In M. Murtonen, J. Rautopuro, & P. Väisänen (Eds.), Learning and teaching of research methods at university (pp. 163–184). Finnish Educational Research Association.

483

Murtonen, M. (2015). University students’ understanding of the concepts empirical, theoretical, qualitative and quantitative research. Teaching in Higher Education, 20(7), 684–698. https://doi.org/ 10.1080/13562517.2015.1072152 Niglas, K. (2007). Introducing the quantitative-qualitative continuum: An alternative view on teaching research methods courses. In M. Murtonen, J. Rautopuro, & P. Väisänen (Eds.), Learning and teaching of research methods at university (pp. 185–203). Finnish Educational Research Association. Niglas, K. (2009). How the novice researcher can make sense of mixed methods designs. International Journal of Multiple Research Approaches, 3(1), 34–46. https://doi.org/10.5172/mra.455.3.1.34 Noble, S. U. (2018). Algorithms of oppression: How search engines reinforce racism. New York University Press. Olkin, R., Hayward, H., Abbene, M. S., & VanHeel, G. (2019). The experiences of microaggressions against women with visible and invisible disabilities. Journal of Social Issues, 75(3), 757–785. https://doi.org/10.1111/josi.12342 Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in the Schools, 13(1), 48–63. www.msera.org/docs/ rits-v13n1-complete.pdf#page=55 Onwuegbuzie, A. J., & Leech, N. L. (2005). Taking the “Q” out of research: Teaching research methodology courses without the divide between quantitative and qualitative paradigms. Quality & Quantity, 39(3), 267–295. https://doi.org/10.1007/s11135004-1670-0 Patel, S. G., Clarke, A. V., Eltareb, F., Macciomei, E. E., & Wickham, R. E. (2016). Newcomer immigrant adolescents: A mixed-methods examination of family stressors and school outcomes. School Psychology Quarterly, 31(2), 163–180. https://doi. org/10.1037/spq0000140 Pauwels, L. (2010). Visual sociology reframed: An analytical synthesis and discussion of visual methods in social and cultural research. Sociological Methods & Research, 38(4), 545–581. https://doi. org/10.1177/0049124110366233 Plano Clark, V. L., & Sanders, K. (2015). The use of visual displays in mixed methods research: Strategies for effectively integrating the quantitative and qualitative components of a study. In M. T. McCrudden, G. Schraw, & C. W. Buckendahl (Eds.), Use of visual displays in research and testing: Coding, interpreting, and reporting data (pp. 177–206). Information Age Publishing. Plano Clark, V. L., & Ivankova, N. V. (2016). Mixed methods research: A guide to the field. Sage. Poth, C. (2014). What constitutes effective learning experiences in a mixed methods research course? An examination from the student perspective.

484

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

International Journal of Multiple Research Approaches, 8(1), 74–86. https://doi.org/10.5172/ mra.2014.8.1.74 Robinson-Wood, T., Balogun-Mwangi, O., Boadi, N., Fernandes, C., Matsumoto, A., Popat-Jain, A., & Zhang, X. (2015). Worse than blatant racism: A phenomenological investigation of microaggressions among Black women. Journal of Ethnographic & Qualitative Research, 9(3), 221–236. www.jeqr.org/ previous-volumes-1/volume-9-issue-3 Shah, P., & Hoeffner, J. (2002). Review of graph comprehension research: Implications for instruction. Educational Psychology Review, 14(1), 47-69. https://doi.org/10.1023/A:1013180410169 Shannon-Baker, P. (2021). The palimpsest as an artsbased integration strategy for mixed methods research. Caribbean Journal of Mixed Methods Research, 2(1), 1–19. https://doi.org/10.37234/ CJMMR.2021.0201.A01

Shannon-Baker, P. (2022). Virtual Special Issue on “Mixed methods designs, integration, and visual practices in educational research.” Journal of Mixed Methods Research, 16(2), 159–164. https:// doi.org/10.1177/15586898221083959 Shannon-Baker, P., & Edwards, C. (2018). The affordances and challenges to incorporating visual methods in mixed methods research. American Behavioral Scientist, 62(7), 935–955. https://doi. org/10.1177/0002764218772671 Tashakkori, A., & Teddlie, C. (2003). Issues and dilemmas in teaching research methods courses in social and behavioural sciences: US perspective. International Journal of Social Research Methodology, 6(1), 61–77. https://doi.org/10.1080/13645570305055 Webb, R. B. & Glesne, C. (1992). Teaching qualitative research. In M. D. LeCompte, J. Preissle, & W. L. Millroy (Eds.), The Handbook of Qualitative Research in Education (pp. 771–814). Academic Press.

33 Towards the Future Legitimacy of Mixed Methods Designs: Responsible Mixed Methods Research for Tackling Grand Challenges for the Betterment of Society J o s é F. M o l i n a - A z o r i n a n d M i c h a e l D . F e t t e r s

INTRODUCTION The world is confronted with grand challenges. Grand challenges are formulations of global problems that can be plausibly addressed through coordinated and collaborative effort (George et  al., 2016). Grand challenges have a high level of complexity and uncertainty. In this regard, Mertens (2015) uses the term “wicked problems”, emphasizing that these problems are replete with social and institutional uncertainties. Examples of grand challenges include the COVID-19 pandemic, poverty, inequalities, human rights violations, migration, environmental degradation and climate change. Arguably, the goal of science is to seek truth, and the ultimate purpose of seeking truth is to provide valid explanations about empirical phenomena to make improvements in the natural and social worlds (Tsui, 2013). Socially conscious scholars must encourage studies on societal problems and aspire to join global efforts at understanding and solving societal grand challenges (George et al., 2016; Howard-Grenville et al., 2019; MolinaAzorin & Fetters, 2019). As we examine in this chapter, mixed methods research has the potential for addressing the complexity of grand challenges.

In the social sciences, credibility and legitimacy of research represent important issues of debate. For example, in the management field, some critical observers have noted that researchers usually do not conduct credible and relevant research that can help to solve societal problems (Harley, 2019; Wickert et al., 2021). A key problem is that sometimes researchers examine topics that are not relevant and important for practitioners and society. A central premise of this chapter is that grand challenges provide an opportunity and a catalyst for transformation. These societal challenges can be a guide to scholars who want to make a difference to promote a better society. Through research and education, mixed methods scholars are positioned uniquely to make a difference by providing critical stakeholders with the information needed to address and solve societal issues. The purpose of this chapter is to examine how mixed methods designs can help understand and address current and future societal problems. We consider this as a key aspect for the future legitimacy of our mixed methods community. The mixed methods research community needs to conduct research that serves society. In other words, the future legitimacy of mixed methods research

486

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

designs depends on their usefulness to address and solve grand societal challenges. In our view, scholars can improve not only their academic impact, but also their practical and social impact through use of mixed methods research. Mixed methods researchers are well positioned to make an intellectual contribution to better understand societal challenges, their main causes and consequences, and to identify solutions. By adopting a transformative (Chapter 4, this volume) and responsible approach to research, mixed methods researchers may play a key role in tackling societal problems. In this chapter we examine issues about what constitutes impactful research and the responsibility of social researchers to be accountable for advancing the field relative to the grand challenges faced. We review principles of responsible research that may guide studies and describe how mixed methods designs can use these principles to address grand societal challenges. Finally, we examine several empirical studies that address timely grand challenges using mixed methods research.

RISING TO THE OCCASION OF GRAND CHALLENGES: IMPACTFUL AND RESPONSIBLE APPROACHES FOR THE BETTERMENT OF SOCIETY In this section, we argue for a wide perspective of the term “impact” beyond traditional metrics linked to research productivity in terms of number of publications. An emphasis on practical and societal impact must be considered. In addition, we examine how principles of responsible research can guide mixed methods researchers to conduct studies that help address grand challenges.

Reconsidering Assessment of the Impact of Social Research In several fields, researchers are debating about the meaning of the term “impact” and how to determine the characteristics of impactful research rather than just the number of publications in top journals and citations (Aguinis et al., 2021; Wickert et al., 2021). For example, in the field of management, scholars have proposed for the assessment of impact the notion of “social impact” in the group of characteristics that management scholars should consider for conducting impactful research. Wickert et  al. (2021) examined five forms of impact, emphasizing how management scholars can extend or enlarge their research agenda to amplify their impact on societal challenges.

Together with scholarly impact (the ability to provide a clear and meaningful theoretical contribution), practical impact (focused mainly on management practices for financial performance optimization) and educational impact (how research might impact education), these authors explicitly add two additional impacts that focus on societal challenges: societal impact (how scholars can contribute to societal concerns) and public policy impact (how scholars can provide a deeper understanding of important policy issues among political decision-makers). In this regard, grand societal challenges may help motivate how management scholars speak to broad social and ecological issues facing global societies. Aguinis et  al. (2021) also propose a multidimensional and multistakeholder model for characterising impactful research. They include four dimensions: (1) the theory and research dimension, where the key stakeholders are other researchers and impact is usually measured using citations; (2) the education dimension, which emphasizes the aspiration to have an impact on students; (3) the organizations’/practitioners’ dimension that denotes the influence on managers and consultants who use the knowledge generated by research; and (4) the society dimension, where the goal is to influence policy makers to solve societal problems. Therefore, the impact of our studies must be broadened to include not only the number of publications and citations, but also practical and societal impact. This wide definition of impactful research has implications for research methods, and specifically for mixed methods research. Next, we examine the application of several principles of credible and relevant research to mixed methods research designs.

Principles of Responsible Research: Bringing Attention to Practical and Relevant Social Research in Mixed Methods Congruous with this expanding view of research impact assessment that goes beyond the number of publications in top journals and citations, there are several movements and initiatives aiming to promote social research that is practical and relevant. As indicated in an editorial in the Journal of Mixed Methods Research (Molina-Azorin & Fetters, 2019), we noted an important initiative in the management field known as the Community for Responsible Research in Business and Management. In a position paper (Co-founders of Responsible Research in Business and Management, 2017),

TOWARDS THE FUTURE LEGITIMACY OF MIXED METHODS DESIGNS

this community articulated seven principles for conducting responsible science that were designed to address problems from a crisis of confidence in management studies through the production of “credible” and “useful” knowledge that addresses problems that are important to business and society. In this regard, “credible” knowledge is created with scientific rigour. And “useful” knowledge is considered from a broad perspective, taking into account two key points: first, knowledge must be relevant and actionable for better management practices in organizations; and second, knowledge

487

must help address societal problems. Therefore, responsible research aims to address problems important not only to business but also to society. In our opinion, these principles are valid not only for management researchers, but also for other social scholars and specifically for the mixed methods research community. We have adapted these seven principles. The updated and adapted principles are summarized in Table 33.1. These recommendations largely reflect the content of the original principles, but have been changed in three important ways. First,

Table 33.1  Eight principles for guiding responsible research in the field of mixed methods research Principle

Explanation

(1) Serve society

Mixed methods researchers, teachers and organizational leaders aim to develop knowledge that advances the social sciences and broader society, both locally and globally, for the ultimate purpose of creating a better world. Mixed methods researchers, teachers, journal editors, funders, accrediting agencies and other stakeholders will respect and recognize contributions in both theoretical and applied mixed methods research. Mixed methods researchers, teachers, organizational leaders, journal editors, funders and accreditation agencies value diversity in research themes, methods, forms of scholarship, types of enquiry and interdisciplinary collaboration to reflect the plurality and complexity of mixed methods research for addressing societal problems. Mixed methods researchers will utilize, implement and advance rigorous scientific methods and processes in integrating quantitative and qualitative methodology. Mixed methods researchers, teachers and organizational leaders value the involvement of different stakeholders who can play a critical role at various stages of the scientific process, without compromising the independence or autonomy of enquiry. Mixed methods researchers, teachers and organizational leaders, funders, and accrediting agencies, support, acknowledge and reward research that has an impact on diverse stakeholders, especially research that contributes to a better world. Mixed methods researchers, teachers and organizational leaders will conduct and support research that shows respect for persons, social justice and environmental sustainability for stakeholders. Mixed methods researchers, teachers and organizational leaders value diverse forms of knowledge dissemination to collectively advance basic knowledge and practice of multiple stakeholders at multiple levels of society.

(2) Value both basic and applied contributions

(3) Value plurality and multidisciplinary collaboration

(4) Use sound methodology

(5) Involve stakeholders

(6) Emphasize research with an impact on stakeholders

(7) Conduct research ethically

(8) Disseminate broadly the findings to research stakeholders

Source: Adapted from Co-founders of Responsible Research in Business and Management (2017).

488

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

the principles were rewritten with verbs as the original principles included nouns. Using verbs in the principles clarifies actively operationalizing the stated principles. Second, by changing the perspective from management, business leaders and educators to the mixed methods research community that we define as mixed methods researchers, teachers and organizational leaders, in addition to journal editors, funding organizations, accrediting organizations and other key stakeholders that had already been included. Third, we added an additional principle represented by Principle 7 in this list that addresses the ethical conduct of research. While Principle 4 emphasizes using sound methods—research conducted with rigorous integrity—the original seven principles lacked an explicit statement about the importance of ethical conduct of social research. The emphasis of the added principle includes respect for persons, seeking social justice and striving for environmental sustainability. Principle 1 is a general principle that emphasizes that science should bring benefits to society and should avoid negative externalities. Principles 2 to 4 try to improve the credibility, reliability and rigour of research studies, encouraging plurality, multidisciplinary collaboration, contexts and sound methodology. Principles 5– 8 highlight the relevance of discoveries, ethical conduct of research and the accessibility of research findings to stakeholders and end-users. The application of these principles may promote the use of mixed methods research to understand and address grand societal challenges, improving the impact of our mixed methods studies. Next, we examine how mixed methods research can contribute to the betterment of society through transformative research, considering these principles of responsible research.

TRANSFORMATIVE RESEARCH AND APPROACHES CONDUCIVE TO RESPONSIBLE INVESTIGATIONS THAT CAN BE CONSIDERED BY MIXED METHODS RESEARCHERS A transformative approach to research may help to support and facilitate the application and implementation of these principles of responsible research. The transformative research lens incorporates ideas such as consciously addressing power differences with strategies that allow for the inclusion of the voices of the full range of stakeholders (Mertens, 2007, 2009, 2012, 2018,

2020; Mertens & Wilson, 2019; see also Chapter 2, this volume). The main goal of the transformative approach is to support the development of culturally responsive interventions that foster increased respect for human rights and achievement of social, economic and environmental justice (Widianingsih & Mertens, 2019). Therefore, through this transformative approach and appropriate methods that promote this perspective, scholars may conduct research that addresses grand societal challenges. Donna Mertens, in her books and articles cited above, underlines three basic beliefs associated with this transformative approach. First, regarding ontology, the transformative approach considers that there are multiple realities that are socially constructed, being necessary to be explicit about the social, political, cultural, economic, racial and gender values that define realities. Second, with regard to epistemology, to know realities it is necessary to have an interactive link between the researcher and the participants in a study as knowledge is socially located within a complex cultural context. Third, from the methodology perspective, a researcher can choose quantitative, qualitative or mixed methods, but there should be an interactive relationship between the researcher and the relevant stakeholders in the definition of the problem, and methods should be adjusted to accommodate cultural complexity. As noted in these characteristics of the transformative approach to research and in principle 5 of responsible research, involvement of stakeholders is essential for addressing grand challenges. In this regard, these stakeholders and participants can play a critical role at various stages of the research process: from the formulation of the key problems to solve and research questions to answer in the specific context, to the interpretation of the main findings and identification of actions to solve the problems. This involvement of practitioners and community members in several stages of the scientific process may help emphasize the usefulness and relevance of our research to solve grand challenges. Therefore, the identification of a relevant social question that matters for these stakeholders is an important step (Vermeulen, 2005). Another key characteristic of the transformative approach and responsible research is that research should impact and lead to actionable outcomes for the practitioners, stakeholders or the community members (principle 6). Then, we should consider research pathways that lead to actionable impact (Voss, 2020). Research methods provide the means for researchers to carry out impactful, responsible and transformative research to address grand challenges. As noted in the Responsible Mixed

TOWARDS THE FUTURE LEGITIMACY OF MIXED METHODS DESIGNS

Methods Research Principles (Table 33.1), the use of sound methodology and the value of a diversity of methods provide guidance for conducting responsible research. An important feature of grand challenges is their multilevel nature because these social problems implicate actions at multiple levels, namely, the individual, the organizational and the institutional levels. Additionally, they manifest at local, regional, national and global scales (George et  al., 2016; Howard-Grenville et  al., 2019). Moreover, grand challenges require novel ideas and unconventional research approaches to tackle their complex and evolving mix of technical and social elements. In this context, mixed methods research can play a key role. Social science researchers have the opportunity to consider a diversity of methods that may help address grand challenges of society. Mixed methods research by definition involves diverse methods, both qualitative and quantitative that extend multiple and diverse use of methods to operationalizing through integration in multiple dimensions of the research enterprise. In an editorial in the 2019 July issue of the Journal of Mixed Methods Research (Molina-Azorin & Fetters, 2019), we highlighted the leadership role of Donna Mertens advocating for a transformative approach and the strength of mixed methods as a force for achieving social change. Moreover, we indicated that the Sustainable Development Goals (SDGs) serve as a list of problems that mixed methods researchers can address, and these qualify as grand challenges that must be addressed to build a better world. In this editorial, we emphasized that mixed methods researchers are positioned to address societal problems, to make a difference and to resolve grand challenges by (1) using a complete methodological toolkit; (2) integrating expertise across other diverse research methodologists; (3) engaging stakeholders and involving them in the creation of knowledge; (4) producing evidence that resonates; and (5) disseminating, evaluating and demonstrating the impacts of their research (Molina-Azorin & Fetters, 2019). Qualitative research methodologists often use the methodology to give voice to those who would not be heard. By using qualitative methodology in mixed methods studies, mixed methods researchers can leverage the qualitative strand of mixed methods design to promote contact and conversations with several stakeholders at different stages in the research process, and this approach can facilitate the identification of questions and issues that are interesting and relevant for these stakeholders in their context and their view of society. In consideration of the principles in Table 33.1, these tenants are consistent with Responsible Mixed Methods Research principle 5 of involving

489

stakeholders in the research, and principle 6 of rewarding research that has an impact on diverse stakeholders. These points suggest that an important element of responsible and transformative research is the co-creation of knowledge with stakeholders (Sharma & Bansal, 2020) and how mixed methods research can help in this process. For example, in the field of management, researchers attempt to impact practice and society but they often fail to do so. This aspect is known as the “rigour-relevance gap” (rigour considered as the credibility and reliability of research, and relevance considered as the impact of research on managers and society). Simply stated, overemphasis on rigour can come at the cost of relevance of the findings. Management scholars have noted that researchers and managers belong to different knowledge communities and use different knowledge systems. While researchers often seek rigour through generalizable insights, managers, policy makers and other practice stakeholders prefer relevant and context-specific advice. As indicated by Sharma and Bansal (2020), through the processes of co-creation of knowledge, researchers and relevant stakeholders can share insights, spend time together to jointly determine interesting and relevant research questions, and interpret data to identify actions to solve practical and societal issues. Other interesting approaches conducive to responsible and transformative investigations through mixed methods studies are communitybased participatory research and participatory action research. Community member involvement and participation in the research process can synergize the co-creation of knowledge. In this regard, community-based participatory research (Israel et al., 2013) is an approach to research that equitably engages communities in the design and implementation of research that benefits the local setting. Community members, stakeholders and practitioners are fully involved as research partners who engage in shared decision making and co-research together with scholars. Communitybased participatory researchers commonly combine a mixed methods approach with the communitybased participatory research methodology as noted in the DeJonckheere et al. (2019) methodological review that identified studies using mixed methods community-based participatory research. Through this methodological approach, research findings can benefit community members through community empowerment, through an approach where their needs are prioritized and addressed, through tangible benefits of co-created interventions, and through improved research skills and capacity. Similarly, participatory action research using a mixed methods approach (Ivankova, 2015)

490

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

also provides a framework for conducting responsible and transformative research. Participatory action research is based on a collaborative problem-solving relationship between researchers and practitioners which aims at both solving a problem and generating new knowledge. The process of co-creation of knowledge can also be implemented with students as the key stakeholders. For an illustrative example of students as co-researchers in game-based research in mixed methods, see Chapter 24 (this volume). In summary, mixed methods research and its scaffolding (Fetters, 2020) or intersection (Plano Clark & Ivankova, 2016) with other methodological approaches can be used to conduct responsible and transformative research with a broader societal impact that can contribute to the betterment of society. Having established the premise for socially impactful, transformative and responsible research, and approaches for conducting mixed methods research through participatory designs, in the following, we describe several mixed methods studies that have addressed grand challenges to build a better world.

EXAMPLES OF MIXED METHODS STUDIES TACKLING GRAND CHALLENGES Addressing the grand challenge of poverty, unemployment and environmental degradation Widianingsih and Mertens (2019) applied a mixed methods transformative approach in West Java, Indonesia, which is a region of the world with conditions of serious poverty, unemployment and environmental degradation. In their efforts to achieve the Sustainable Development Goals, their use of the transformative research approach involved conducting a contextual analysis to understand the challenges in the research context and address power differences with strategies that allowed for the inclusion of the voices of the full range of stakeholders. The viewpoint of stakeholders from different parts of society was meant to bring visibility to the marginalized and give them a voice. They included people from rural areas, women, youth and people with disabilities. Using data from several stakeholders, the researchers contrasted the views of those in positions of power with those who were not in formal positions of power. As explained by the authors, they used several phases in this transformative mixed methods

research project. The project began with a phase of relationship building to identify relevant stakeholders and develop strategies for working together. In the next phase, they conducted a contextual analysis that involved collection of quantitative and qualitative data to elucidate a better picture of the economic, political and demographic variables relevant to the identified context. In the following phase, data from the previous phases were used to inform decision makers and communities about potential interventions to be pilot tested, that could be later implemented through a plan to support change.

Addressing the grand challenge of equity promotion for marginalized populations In reflecting about the application of mixed methods research with groups at risk based on a special issue published in the Journal of Mixed Methods Research, Sorde Marti and Mertens (2014) identified two main characteristics of mixed methods research highlighting its potential contributing to social change. They advocated for a transformative approach and involvement of the most underserved communities in the research process. In the context of research promoting equity for marginalized populations, as the first characteristic, they emphasized the responsibility of scholars to not only make progress in the diagnosis and understanding of social problems, but also to inform actions or policies through scientific evidence that has the purpose of solving and reversing the problems. Underserved communities and policy makers can consider mixed methods studies that examine existing cases of transformation. As the second characteristic, they emphasize involving the communities and people who are being researched. This requires not only including the voices of the researched subjects in the process of investigation, but also a progression from research “on them” to “with them”. Therefore, these authors highlight carrying out a process of cocreation of knowledge with relevant stakeholders and illustrate key principles of responsible research as noted previously in this chapter. As one example from that special issue, Zea et  al. (2014) engaged in a research with a group that has suffered greatly from violence and marginalization: Colombian gay and bisexual men and transwomen. Using the theory of communicative action, the authors described their approach using mixed methods research to support social change through giving voice and visibility to the group. The gay and bisexual men and transwomen were

TOWARDS THE FUTURE LEGITIMACY OF MIXED METHODS DESIGNS

viewed as legitimate actors with valuable knowledge to share. In the qualitative strand of the study, the authors conducted life history interviews and illustrated due consideration of communicative action—namely, non-coercive dialogue, which contributed to greater knowledge of the experiences and social perspectives of the group. The quantitative component was conducted through a survey. As indicated by the authors, the use of a mixed methods approach afforded a rich perspective of the lived experiences of participants. For example, regarding sex work, the quantitative component identified displacement as a driving force motivating many individuals to engage in sex work, whereas qualitative data provided a nuanced understanding of the experience of sex work, including negative and positive aspects. The authors also emphasized the importance of dissemination of the findings among members of the study community, as well as among people in power to effect change at the policy level. The issue of dissemination is a key principle of responsible research, as identified by the Co-founders of Responsible Research in Business and Management (2017) as introduced previously in this chapter. Scholars can also use secondary data for researching topics related to marginalized groups. For a discussion of integrating secondary data from ethnically and racially marginalized groups into mixed methods research, see Chapter 25 (this volume).

Addressing the grand challenge of migration For a special issue on migration, in the editorial in the Journal of Mixed Methods Research introducing the articles, Bergman (2018) reflected how mixed methods research is well positioned to address the grand challenge of migration as it is suited for dealing with complexity, mutability and transdisciplinary research that are all characteristic problems associated with migration. Bergman emphasized that mixed methods designs can capture the multiplicity of phenomena associated with migration, and elucidate solutions and policies for relevant stakeholders. In a commentary, Horvath and Latcheva (2019) noted the many methodological challenges linked to migration research, and note that these challenges are not only about the complexity of cross-border mobility, but also the political dynamics that have affected migration practices. They emphasized the importance of migration researchers being aware of the political entanglements of their studies, and the value of the mixed methods approach to address the double methodological challenge. They also emphasized

491

the need for researchers listening to and taking account of subordinated voices initially to frame the research problems. This example provides another illustration of co-creation of knowledge in facing grand challenges. Among several examples of methodological innovations featured in the migration research special issue, the article by Kaczmarczyk and Salamonska (2018) focused on using ethnosurvey methodology in a study about migration in Poland. Ethnosurvey is a combination of ethnography and survey research methodologies. The methodological approach can be used in migrant origin and destination countries with the purpose of a better understanding of multiple facets and the whole complexity of migration processes. In the Polish case, early approaches and use of ethnosurveys were driven by the need to collect valid statistical data on migration. For that purpose, the quantitative part was more valued. But the qualitative strand provided more in-depth information about other important features of migration: the nature of mobility, main drivers of migration, and the main consequences of the migration process. The authors used integration strategies for various dimensions of their research such as the instrument design, data collection, data analysis and interpretation. Ethnosurvey methodology is a hybrid of two methodologies that is an inherently mixed methods approach positioning researchers to address the grand challenge of immigration.

Addressing the grand challenge of the COVID-19 pandemic The Journal of Mixed Methods Research also published a special issue on the Covid-19 pandemic. The call for papers cited unprecedented need for mixed methods and advances in methodological innovation to use and create novel mixed methods methodologies for addressing this pandemic and to inform future catastrophic social changes (Fetters & Molina-Azorin, 2020). With seven articles on methodological innovations relative to the COVID-19 pandemic, the special issue served as a milestone for advancing mixed methodology (Fetters & Molina-Azorin, 2021). With the pandemic and its aftermath still far from over, the mixed methods community must continue to innovate for the grandest of contemporary grand challenges. Among several articles in the special issue, the example by Poth et al. (2021) showcased a number of methodological innovations to address the grand challenge of the COVID-19 pandemic. The overarching purpose of the study was to analyze

492

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

public health briefings to examine how these briefings built credibility and trust for effective public health communication during the changing conditions of the COVID-19 pandemic in Alberta (Canada). The authors employed “a complexity-informed case study mixed methods research design with a nested convergent sequential approach”. The authors used complexity theory to inform the study by examining three central concepts of complex adaptive systems— namely, emergence, interdependence and adaptation. The authors characterized the overall design as a case study mixed methods design which they appropriately bounded to their province, the office communications of the province’s chief medical officer, and a specific time frame. Mixed methods data collection was conducted sequentially where an initial strand of quantitative research was conducted using sentiment analysis based on quantitative text mining and qualitative thematic analysis of the public health briefings. This was followed by a convergent strand integrating quantitative case statistics, themes from the briefings, and a key event timeline. Sentiment analysis in the initial quantitative strand provided a way to detect negative and positive patterns within the large text database of media briefings. Atypical periods of sentiment fluctuation in the first phase were used to identify six fluctuation periods of integration within the second phase of the mixed methods design that were characterized using findings from four data sources: qualitative themes from the media briefing transcripts, key timeline events from the online news searches, key topics and associated terms from the quantitative topic modelling, and new case statistics from the quantitative descriptive analysis. The study contributed to public health communication by illustrating how trust was built, credibility was enhanced, and functions of public health officials enhanced risk communication, showing empathy and providing updated information.

CONCLUSIONS: A LOOK FORWARD A challenge for conducting rigorous and relevant mixed methods studies is to provide transparency about how and where integration of quantitative and qualitative methods is achieved. Fetters et al. (2013) and Plano Clark (2019) provide important ideas about how to achieve integration in mixed methods research. Guetterman et  al. (2015) and Plano Clark and Sanders (2015) examine an important innovation in mixed methods: how the use of visual joint displays (tables, graphic

displays) may facilitate the integrative process. In this regard, technological advances and software applications may be useful. For example, Guetterman et  al. (2015) discuss relevant joint displays that may be useful for specific mixed methods designs, indicating procedures to create each type of joint display using a specific computer application, MAXQDA. Mertens et  al. (2016) also refer to the relevance of technological advances. Gobo et al. (2021) examine opportunities for integrating methods using computational techniques, such as text mining. Therefore, technological and procedural advances may play a key role in the next years to take advantage of synergies and the potential of integration in mixed methods research to address grand challenges. In consideration of principles of responsible research, mixed methods researchers are primed to make a difference, and to contribute to the understanding and resolution of societal problems through integration of quantitative and qualitative methods. With mixed methods findings, researchers can equip diverse stakeholders to tackle the Sustainable Development Goals (SDGs) (Molina-Azorin & Fetters, 2019) and other grand societal challenges. An important challenge, but also opportunity, is that the mixed methods community needs to accept responsibility as social scientists. For achieving this purpose, a key premise is that the mixed methods community needs to shift, or at least to complement, the role as knowledge producers by emphasizing our role as activists and social change agents with an ethical responsibility (Chapter 4, this volume). A transformative approach to research may help to support and facilitate the role and attitude of mixed methods scholars as activists and social change agents. A starting point comes from the researchers’ intrinsic desire as scholars to make a difference for the betterment of society by envisioning their playing an activist role. If researchers adopt this attitude through passionate scholarship (Courpasson, 2013), the commitment becomes personally meaningful and the topic socially relevant. As argued by previous scholars (Contu, 2020; Wickert & Schaefer, 2015), researchers need to develop this attitude of intellectual activism and critical performativity in order to make a difference in the world. As noted above, the transformative approach through mixed methods research can serve as a model for researchers taking an activist role. Greene (2007) pointed out that a dialectical stance in mixed methods research enables engagement with difference and promotes values of tolerance, acceptance and equity. Moreover, as Greene emphasized, values are interwoven with

TOWARDS THE FUTURE LEGITIMACY OF MIXED METHODS DESIGNS

methodologies and social science research can position their work in service of the public good. In this chapter, we have examined a broad perspective of impact, the principles of responsible research, the potential of transformative research and some examples of mixed methods studies that address important grand challenges. We hope to have motivated mixed methods researchers to consider the opportunities for making an impact for a better society, trying to understand and solve grand societal challenges together with practitioners and key stakeholders. The future we envision for mixed methods research designs is its key role to conduct credible, rigorous, useful and relevant research that helps address and solve the main problems and challenges of our society. Basic research is important, but we also need applied research that provides actionable solutions through responsible and transformative co-creation of knowledge with stakeholders. We need mixed methods research designs that provide solutions based on evidence (credible and relevant evidence based on the principles of responsible research), solutions developed with involvement of people (collaboration of mixed methods researchers and relevant stakeholders), and taking advantage of available technology. These aspects will be key features of research needed to increase legitimacy of our mixed methods field for the next decades. Through capacity building with appropriate training of scholars on mixed methods research designs, we will be able to develop this vision of serving society, as indicated in the first principle of responsible research. Therefore, the impact of our research must be broadened to include not only academic impact (number of publications and citations) but also societal impact. We need to assume the responsibilities that come with the power that we possess as scholars. As examined in this chapter, a mixed methods approach can help us in this endeavour.

WHAT TO READ NEXT Mertens, D. M. (2009). Transformative research and evaluation. Guilford Press. Mertens, D.M. & Wilson, A.T. (2019). Program evaluation theory and practice: A comprehensive guide (2nd ed.). Guilford Press.

Donna Mertens has advocated for the use of mixed methods designs as a force for societal change through transformative research. These two books and Chapter 4 in this Handbook

493

indicate the main ideas of transformative research, providing guidance to design and implement transformative studies. Molina-Azorin, J.F., & Fetters, M. (2019). Building a better world through mixed methods research. Journal of Mixed Methods Research, 13, 275-281. https://doi.org/10.1177/1558689819855864

In this editorial of the Journal of Mixed Methods Research, we examine the role of mixed methods designs for promoting responsible research to address social problems.

REFERENCES Aguinis, H., Yu, L., & Tosun, C. (2021). How to enhance scholarly impact: Recommendations for university administrators, researchers and educators. International Journal of Contemporary Hospitality Management, 33(8), 2485–2499. doi: 10.1108/IJCHM-10-2020-1189 Bergman, M. M. (2018). The century of migration and the contribution of mixed methods research. Journal of Mixed Methods Research, 12, 371–373. https://doi.org/10.1177/1558689818801737 Co-founders of Responsible Research in Business and Management (2017, revised 2020). A vision of responsible research in business and management: striving for useful and credible knowledge. Retrievable from www.rrbm.network. Contu, A. (2020). Answering the crisis with intellectual activism: Making a difference as business schools scholars. Human Relations, 73, 737–757. https://doi.org/10.1177/0018726719827366 Courpasson, D. (2013). On the erosion of ‘passionate scholarship’. Organization Studies, 34, 1243–1249. https://doi.org/10.1177/0170840613502292 DeJonckheere, M., Lindquist-Grantz, R., Toraman, S., Haddad, K., & Vaughn, L. (2019). Intersection of mixed methods and community-based participatory research: A methodological review. Journal of Mixed Methods Research, 13, 481–502. https:// doi.org/10.1177/1558689818778469 Fetters, M. D. (2020). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Sage. Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs: Principles and practices. Health Services Research, 48(6): 2134–2156. https://doi.org/10.1111/1475-67 73.12117 Fetters, M. D., & Molina-Azorin, J. F. (2020). Call for papers for a special issue on Covid-19 and novel mixed methods methodological approaches during catastrophic social changes. Journal of Mixed

494

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Methods Research, 14, 281–287. https://doi.org/ 10.1177/1558689820920098 Fetters, M., & Molina-Azorin, J. F. (2021). Special issue on Covid-19 and novel mixed methods methodological approaches during catastrophic social changes. Journal of Mixed Methods Research, 15, 295–303. https://doi.org/10.1177/1558689 8211029100 George, G., Howard-Grenville, J., Joshi, A., & Tihanyi, L. (2016). Understanding and tackling grand challenges through management research. Academy of Management Journal, 59, 1880-1895. https:// doi.org/10.5465/amj.2016.4007 Gobo, G., Fielding, N. G., La Rocca, G., & van der Vaart, W. (2021). Merged methods. A rationale for full integration. Sage. Greene, J. C. (2007). Mixed methods in social inquiry. Jossey-Bass. Guetterman, T., Creswell. J. W., & Kuckartz, U. (2015a). Using joint displays and MAXQDA software to represent the results of mixed methods research. In M. T. McCrudden, G. Schraw, & C. W. Buckendahl (Eds.), Use of visual displays in research and testing: Coding, interpreting, and reporting data (pp. 145–175). Information Age Publishing. Guetterman, T., Fetters, M. D., & Creswell, J. W. (2015b). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm. 1865 Harley, B. (2019). Confronting the crisis of confidence in management studies: why senior scholars need to stop setting a bad example. Academy of Management Learning & Education, 18, 286–297. https://doi.org/10.5465/amle.2018.0107 Horvath, K., & Latcheva, R. (2019). Mixing methods in the age of migration politics: a commentary on validity and reflexivity in current migrations research. Journal of Mixed Methods Research, 13, 127–131. https://doi.org/10.1177/1558689818822445 Howard-Grenville, J., Davis, J., Dyllick, T., Miller, C., Thau, S., & Tsui, A. (2019). Sustainable development for a better world: contributions of leadership, management and organizations. Academy of Management Discoveries, 5, 355–366. https://doi. org/10.5465/amd.2019.0275 Israel, B., Eng, E., Schulz, A., & Parker, E. (2013). Methods for community-based participatory research for health. Jossey-Bass. Ivankova, N. (2015). Mixed methods applications in action research: From methods to community action. Sage. Kaczmarczyk, P., & Salamonska, J. (2018). Why we still need ethnosurveys in migration research: Empirical evidence from Poland. Journal of Mixed Methods Research, 12, 458–476. https://doi. org/10.1177/1558689818796370

Mertens, D. M. (2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research, 1, 212–225. https://doi. org/10.1177/1558689807302811 Mertens, D. M. (2009). Transformative research and evaluation. Guilford Press. Mertens, D. M. (2012). Transformative mixed methods: Addressing inequities. American Behavioral Scientist, 56, 802–813. https://doi.org/10.1177/ 0002764211433797 Mertens, D. M. (2015). Mixed methods and wicked problems. Journal of Mixed Methods Research, 9, 3-6. https://doi.org/10.1177/1558689814562944 Mertens, D. M. (2018). Mixed methods design in evaluation. Sage. Mertens, D. M. (2020). Research and evaluation in education and psychology. Integrating diversity with quantitative, qualitative and mixed methods (5th ed.). Sage. Mertens, D., Bazeley, P., Bowleg, L., Fielding, N., Maxwell, J., Molina-Azorin, J. F., & Niglas, K. (2016). The future of mixed methods: A five year projection to 2020 (Task Force Report of the Mixed Methods International Research Association). Retrieved from: https://mmira.wildapricot.org/ resources/Documents/MMIRA%20task%20 force%20report%20Jan2016%20final.pdf Mertens, D. M. & Wilson, A. T. (2019). Program evaluation theory and practice: A comprehensive guide (2nd ed.). Guilford Press. Molina-Azorin, J. F., & Fetters, M. (2019). Building a better world through mixed methods research. Journal of Mixed Methods Research, 13, 275–281. https://doi.org/10.1177/1558689819855864 Plano Clark, V. L. (2019). Meaningful integration within mixed methods studies: Identifying why, what, when, and how. Contemporary Educational Psychology, 57, 106–111. https://doi.org/10.1016/ j.cedpsych.2019.01.007 Plano Clark, V. L., & Ivankova, N. (2016). Mixed methods research. A guide to the field. Sage. Plano Clark, V. L., & Sanders, K. (2015). The use of visual displays in mixed methods research: Strategies for effectively integrating the quantitative and qualitative components of a study. In M. T. McCrudden, G. Schraw, & C. W. Buckendahl (Eds.), Use of visual displays in research and testing: Coding, interpreting, and reporting data (pp. 177–206). Information Age Publishing. Poth, C. N., Bulut, O., Aquilina, A. M., & Otto, S. J. G. (2021). Using data mining for rapid complex case study descriptions: Example of public health briefings during the onset of the Covid-19 pandemic. Journal of Mixed Methods Research, 15, 348–373. https://doi.org/10.1177/15586898211 013925 Sharma, G., & Bansal, P. (2020). Cocreating rigorous and relevant knowledge. Academy of Management

TOWARDS THE FUTURE LEGITIMACY OF MIXED METHODS DESIGNS

Journal, 63, 386–410. https://doi.org/10.5465/ amj.2016.0487 Sorde Marti, T., & Mertens, D. M. (2014). Mixed methods research with groups at risk: New developments and key debates. Journal of Mixed Methods Research, 8, 207–211. https://doi.org/10.1177/155 8689814527916 Tsui, A. (2013). Editorial. The spirit of science and socially responsible scholarship. Management and Organization Review, 9, 375–394. https://doi. org/10.1111/more.12035 Vermeulen, F. (2005). On rigor and relevance: Fostering dialectic progress in management research. Academy of Management Journal, 48, 978–982. https://doi.org/10.5465/amj.2005.19573102 Voss, C. (2020). Toward an actionable and pragmatic view of impact. Academy of Management Discoveries, 6, 538–539. https://doi.org/10.5465/amd. 2019.0113 Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that

495

makes a difference: Broadening the meaning of impact. Journal of Management Studies, 58, 297–320. https://doi.org/10.1111/joms.12666 Wickert, C., & Schaefer, S. M. (2015). Towards a progressive understanding of performativity in critical management studies. Human Relations, 68, 107–130. https://doi.org/10.1177/00187267 13519279 Widianingsih, I., & Mertens, D. (2019). Transformative research and the Sustainable Development Goals: Challenges and a vision from Bandung, West Java. International Journal of Transformative Research, 6, 27–35. https://doi.org/10.1515/ ijtr-2019-0005 Zea, M. C., Aguilar-Pardo, M., Betancourt, F., Reisen, C. A., & Gonzales, F. (2014). Mixed methods research with internally displaced Colombian gay and bisexual men and transwomen. Journal of Mixed Methods Research, 8, 212–221. https://doi. org/10.1177/1558689814527941

34 Realizing Methodological Potentials and Advantages of Mixed Methods Research Design for Knowledge Translation N a t a l i y a V. I v a n k o v a , J a m i L . A n d e r s o n , I v a n I. Herbey, Linda A. Roussel and Daniel Kim

INTRODUCTION An exponential growth in recognition and use of mixed methods research across disciplines and epistemological paradigms points to its methodological potential to support other forms of scientific inquiry (Creswell & Plano Clark, 2018; Plano Clark & Ivankova, 2016), such as translational research (Halcomb, 2019; Ivankova et al., 2018). Translational research focuses on moving knowledge and discovery gained from the basic sciences to its application in professional and community settings and involves multiple processes of knowledge translation from knowledge generation to knowledge adaptation, dissemination, implementation and monitoring (Fort et  al., 2017; Khoury et al., 2010). Despite methodological advances in the knowledge translation process, slow adoption rates and significant delays in translation time call for more effective research designs to optimize translation of evidence-based research findings into everyday practice (Chambers & Norton, 2016; Kajermo et  al., 2010). Mixed methods research that meaningfully integrates qualitative stakeholder engagement methods with quantitative outcome-focused approaches is uniquely positioned to support the knowledge translation process by enhancing the application of multiple methods, thereby providing a way of integrative

thinking for addressing complex knowledge translation problems (Martin, 2007). The critical role of mixed methods in translating knowledge through adaptation, implementation and dissemination research has been well-recognized (Creswell et  al., 2018; Green et  al., 2015; Ivankova et  al., 2023; Ivankova et  al., 2018; Palinkas & Cooper, 2017) aided by discussions of applications of core and advanced mixed methods designs in implementation research (Curry & Nunez-Smith, 2015; Meister, 2018; Palinkas et al., 2011; Shorten & Smith, 2017). While mixed methods designs are commonly used in translational research, their potential has yet to be fully realized for optimizing knowledge translation across disciplines. Knowledge translation is a complex multilevel process consisting of many translational research components and procedures (Straus et al., 2009). Designing effective mixed methods studies for knowledge translation requires considerations of the epistemological, axiological and methodological premises that the intersection of mixed methods and translational research entails. Besides mixed methods design characteristics, researchers should consider the purposes of knowledge translation components, specific uses of mixed methods along the translational research continuum, approaches to evidence-based practices, effective methods to promote stakeholder

REALIZING METHODOLOGICAL POTENTIALS AND ADVANTAGES

engagement and cyclical approaches to solving knowledge translation problems. In this chapter, we address these considerations by discussing how a meaningful intersection of mixed methods with five translational research components at the epistemological, axiological, methodological and design levels provides a methodologically robust approach to designing studies for knowledge translation. We begin with providing brief background information on translational research and knowledge translation and describe the five translational research components for optimizing translational research process with mixed methods designs: evidence-based practice, adaptation, dissemination and implementation, community-based participatory research and action research. Then we discuss a mixed methods framework for translational research and illustrate how we used this framework to elicit the perspectives on designing mixed methods studies for knowledge translation from researchers who actively use mixed methods in a translational research context. We conclude the chapter with discussing additional considerations and challenges and outline future developments in using mixed methods research design for knowledge translation.

TRANSLATIONAL RESEARCH, TRANSLATIONAL SCIENCE AND KNOWLEDGE TRANSLATION Translational research emerged in the medical field stimulated by concerns over the extended time it takes to incorporate scientific discoveries in treatments, practices and health policies (Wethington & Burgio, 2015). It gained increased awareness after the National Institutes of Health’s (NIH) Roadmap initiative in 2006 aimed to bridge the gap between research and practice. Since then, translational research has been used across disciplines to identify, support and promote sustainable adoption of evidence-based practices. Using translational science methods, researchers integrate evidence-based practices, professional experiences and individuals’ preferences to produce outcomes that impact population well-being (Taylor et al., 2016). While translational research and translational science have been used interchangeably to refer to the process of moving knowledge and discovery gained from the basic sciences to its application in professional and community settings (Khoury et al., 2007), there is a distinction between these terms. Translational research is described as a multi-phase and cyclical process (Callard et  al., 2011; Layde

497

et al., 2012), which involves multidisciplinary integration among basic, clinical, practice, population and policy-based research (Zoellner & Porter; 2017). Translational science is defined as the field of investigation which proposes to understand the scientific and operational principles which underscore each step and phase of the translational process (Collins, 2011). The intent of translational science is to advance the process of knowledge translation with results that can have a broader impact by employing effective research designs including mixed methods research (Ivankova et al., 2023). Knowledge translation is both the focus and outcome of translational research. Coined by the Canadian Institutes of Health Research, the term “knowledge translation” implies a dynamic and iterative process of synthesizing, disseminating, exchanging and applying knowledge with the purpose of improving outcomes (CIHR, 2016). It is a complex process of developing, contextualizing and adapting best evidence into practice and requires active engagement of and exchange between researchers and knowledge users (MacDermid & Graham, 2009; Straus et al., 2009). Knowledge translation occurs on a continuum that consists of multiple stages commonly described as the translational science spectrum or translational research continuum (Naylor, 2018). These stages include translating knowledge from the basic sciences to humans, referred to as T1 research; translating research findings to develop new treatments and programmes, referred to as T2 research; translating evidence-based guidelines into everyday practice, referred to as T3 research; and translation to the community and real-world settings, referred to as T4 research (Fort et  al., 2017). These stages have different goals, but they build one upon another and inform other stages in moving the knowledge translation process from discovery to adoption in practice. Designing mixed methods studies in a translational research context requires considerations of the goals of the stages in the translational research continuum and the ways mixed methods research can support these goals by meaningfully intersecting different approaches to generating knowledge and integrating evidence from multiple sources.

METHODOLOGICAL COMPONENTS FOR OPTIMIZING TRANSLATIONAL RESEARCH PROCESS WITH MIXED METHODS DESIGN A complex nature of knowledge translation incorporates both research activities and implementation

498

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

of best practices with users’ input (Layde et  al., 2012; Straus et  al., 2009). Five methodological approaches or components, such as evidence-based practice (EBP), adaptation, dissemination and implementation (D&I), community-based participatory research (CBPR), and action research play an integral role in facilitating translation of research findings to real-world applications (Darwichea et al., 2017; Ivankova et al., 2018; Ivankova et al., 2023). Additionally, these methodological components use mixed methods research to inform a study design and optimize the translational research process by generating evidence grounded in multiple perspectives and reflecting the needs of various stakeholders. In the following sections, we briefly describe these components and their intersection with mixed methods designs. See Ivankova et  al. (2018) and Ivankova et  al. (2023) for a more detailed discussion.

Evidence-based Practice (EBP) EBP is a process of decision-making that integrates the best available evidence, patient/client preferences, and clinical judgment (Melnyk & Fineout-Overholt, 2019). EBP is foundational to translational research and science because it requires objective, balanced and responsible use of relevant research and the best current data to guide scientifically based practice and policy decisions to improve outcomes for consumers (Crime and Justice Institute at Community Resources for Justice, 2009). Mixed methods research enhances EBP by combining consumers’ perspectives, values and experiences with measurable quantitative outcomes to inform improvement decisions. For example, Maloney et al. (2015) used a concurrent mixed methods design to determine the efficacy of social media as an educational medium to effectively translate EBP into clinical settings with the ultimate goal of changing clinicians’ knowledge and behaviours (see also Ivankova et al., 2018).

Adaptation Adaptation is the degree to which an innovation is undergoing adjustments or modifications in the process of its adoption and implementation in real-life settings (Rogers, 1995). Adaptation plays a critical role in translating evidence-based research into practice and increasing the innovation uptake by different people when there is a need to streamline a complex innovation, increase

stakeholders’ ownership of the innovation, address the needs of more diverse populations, choose an appropriate form of innovation delivery or fit the available resources (McKleroy et  al., 2006). Mixed methods research optimizes an adaptation process by allowing researchers to get a better understanding of a particular context, inform cultural adaptations of both content and delivery, and increase the feasibility of the program for the target population. For example, Baydala et  al. (2014) used a concurrent mixed methods design to inform the cultural adaptation of the school-based drug and alcohol abuse prevention program for an Aboriginal community in Central Alberta, Canada (see also Ivankova et al., 2018). For an account of how mixed methods informed the development of the indigenous cultural values instrument in the Malaysian context, see Chapter 14 (this volume).

Dissemination and Implementation (D&I) D&I are active approaches and planned research strategies to promote distribution and use of evidence-based interventions designed for specific audiences or populations (Rabin et  al., 2008; National Institutes of Health, 2021). The aim of D&I research is to facilitate the adoption of EBP in real-world settings by identifying factors and strategies that help reduce time for the innovation uptake (Tabak et al., 2012). Successful D&I of an intervention requires engaging multiple stakeholders, building common goals and ensuring representation throughout the community (de Visser et  al., 2020; Knoepke et  al., 2019), and encourages researchers to use multiple methods to support implementation and evaluation of interventions with wider dissemination efforts (Hull et al., 2019). For example, Roux et al. (2021) used a multi-phase mixed methods design to identify perceived benefits and barriers to implementation of the educational programme to reduce HIV and hepatitis C transmission among people who inject drugs in France (see also Ivankova et  al., 2023). For a discussion of participatory qualitative methods for randomized controlled trials of complex interventions in global health, see Chapter 17 (this volume).

Community-based Participatory Research (CBPR) CBPR is defined as “a systematic inquiry, with the collaboration of those affected by the issue being

REALIZING METHODOLOGICAL POTENTIALS AND ADVANTAGES

studied, for purposes of education and taking action or effecting change” (Green & Mercer, 2001, p. 1927). CBPR is rooted in the idea that understanding the culture and epistemology of a community is key to the enquiry process (DeJonckheere et  al., 2019; Isreal et  al., 2001; Lucero et  al., 2018) and requires partnerships between researchers and community representatives to design and implement studies reflecting community values and needs. Mixed methods research plays a fundamental role in CBPR (DeJonckheere et  al., 2019, Ivankova & Wingo, 2023). Integrating quantitative and qualitative methods in a CBPR study design allows investigators to develop, implement, evaluate and promote interventions that are effective and transferable to other communities and practices. For example, Craig (2011) used a sequential mixed methods design in a CBPR project aimed at developing a system of care for gay, lesbian, bisexual, transgender and questioning (GLBTQ) youths in an urban area (see also Ivankova et al., 2018). For a discussion of leveraging mixed methods CBPR in diverse social and cultural contexts to advance health equity, see Chapter 29 (this volume).

Action Research This cross-disciplinary methodological approach focuses on seeking solutions to practical issues, generating evidence-based knowledge for improving practice, and empowering participants for change action (Hacker, 2013; Stringer & Aragón, 2020). As a systematic and cyclical approach that incorporates both quantitative and qualitative methods, action research consists of four iterative stages of reflecting, planning, acting and observing, and includes six methodological steps: identifying a problem; reconnaissance or fact finding; planning action; acting; evaluating action; and monitoring action (Lewin, 1948). The synergistic combination of mixed methods and action research allows investigators to address practical issues in a systematic and dialectic way by using multiple methods, drawing on multiple stakeholders’ perspectives, and producing more credible and valid results (Ivankova, 2015; Ivankova & Wingo, 2018). For example, Breimaier et al. (2015) used a participatory action research approach combined with a mixed methods concurrent design to identify effective strategies for the implementation by nurses in an Austrian teaching hospital of evidence-based guidelines for preventing falls among older patients (see also Ivankova et al., 2023).

499

Component Synergy in Translational Research Process As shown, each component plays a specific role in facilitating the translational research process, capitalizing on its conceptual and methodological characteristics. However, while pursuing different goals, these components interact and inform each other in a joint effort to promote and facilitate the translation process (Ivankova et  al., 2018). Successful dissemination and implementation of research into practice depend on the nature of ­provided evidence, the contextual environment and the mechanism by which the process is implemented in real-life settings (Kitson et  al., 1998), whereas community and stakeholder engagement is critical for ensuring an evidence-based innovation is accepted and adopted. The process of innovation adaptation is integrated into the D&I process and relies on stakeholders’ involvement and active participation in all stages using CBPR methods (Allen et  al., 2012). Action research with its experiential learning, dynamic nature and focus on the issues relevant to the community provides a foundation for CBPR (Faridi et  al., 2007). The knowledge translation process is often iterative and involves cycles of observing, reflecting, planning and acting to ensure that evidence-based generated findings are reflective of stakeholders’ views and experiences and are aimed at benefitting those affected by the issue (McNiff & Whitehead, 2011).

INTERSECTING MIXED METHODS WITH TRANSLATIONAL RESEARCH FOR KNOWLEDGE TRANSLATION Mixed methods research is uniquely positioned to support a knowledge translation process by meaningfully intersecting with these five components and providing a methodological foundation for integrating quantitative and qualitative methods within a study. When meaningfully combined or intersected with mixed methods research, these methodological components jointly expand the scope of knowledge translation and add rigour to the assessment of the translational research process and outcomes (Ivankova et  al., 2018). Therefore, designing mixed methods studies for knowledge translation requires consideration of the conceptual and methodological features of these components and their meaningful intersection with mixed methods research at the epistemological, axiological, methodological and design levels, as shown in Figure 34.1 (Ivankova &

500

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 34.1  Levels of intersection in a mixed methods translational research study Adapted from Ivankova and Johnson (2022) with permission of Routledge. MMTR: mixed methods translational research

Johnson, 2022; Ivankova & Wingo, 2021; Johnson & Ivankova, 2019). The nested circles capture the interrelationship of different intersection levels in their joint effect on the design of a mixed methods translational research (MMTR) study, which employs pragmatic epistemology, aims at generating and implementing evidence-driven practices, embraces the values of all parties involved in the research process, and integrates multiple quantitative and qualitative methods within a mixed methods design to produce action-oriented inferences. When intersecting at the epistemological level, translational science researchers make decisions about how to generate the types of knowledge to enhance translation and adoption of EBP by capitalizing on the advantages of mixed methods and the components of the translational research process. These decisions relate to the type of research questions to pose, the choice of the methodology and mixed methods design, inclusion of relevant stakeholders, and integration of the quantitative and qualitative methods and data to address these questions. When intersecting mixed methods at the

axiological level, translational science researchers allow different methods to interact within a study by bringing qualitative value-laden stakeholderengagement approaches into a dialogue with and interpretation of the quantitative results (Greene, 2007; Johnson, 2017). Doing so requires careful consideration of the choice of methodology that would allow constructive communication and sharing of values between researchers and study participants. Intersection at the methodological level calls for a systematic and meaningful approach for combining mixed methods with translational research components using a framework to guide methodological decisions related to designing and implementing an MMTR study. A mixed methods action research framework for translational research, that we discuss in the following section, uses community-based participatory action research informed by mixed methods to enhance the processes of innovation adaptation, dissemination and implementation grounded in evidence-based practices. Lastly, intersection at

REALIZING METHODOLOGICAL POTENTIALS AND ADVANTAGES

the design level necessitates translational science researchers to frame their studies in one of the core or complex mixed methods designs (Creswell & Plano Clark, 2018) consistent with the study purpose, translational research goals and anticipated outcomes. As shown in Figure 34.2, the research process in a mixed methods translational research study follows a traditional path of a mixed methods study conceptualization, application of quantitative and qualitative methods and the development of integrated inferences (Tashakkori et al., 2021). Shaped by translational research goals and a practically focused evidence-based improvement problem, a mixed methods study design and selection of methods are influenced by emic perspectives of the stakeholders or individuals who are insiders to the community or organization, and aim at generating action-oriented quality improvementfocused outcomes. Therefore, by meaningfully intersecting with translational research components at the epistemological, axiological, methodological and design levels, a mixed methods approach enhances translational research in several ways. Through the integration of multiple sources of evidence and first-hand experiences, mixed methods research provides a methodological foundation for developing and promoting reliable EBP, generating valid inferences to inform innovation adaptation, securing a systematic approach to innovation exploration, evaluation, dissemination and implementation, and providing a solid ground for

promoting sustainability of the innovation in the community and professional settings.

MMTR METHODOLOGICAL FRAMEWORK The MMTR methodological framework depicted in Figure 34.3 was first described by Ivankova et  al. (2018). The authors adapted the original mixed methods action research framework, developed to facilitate application of mixed methods in community-based action research (Ivankova, 2015) and later applied to promote patient-centeredness (Ivankova, 2017; Ivankova & Wingo, 2021), to the needs of translational research. Since the aim of the original framework is to assist stakeholders in developing better understanding of the data-driven decision-making process by capitalizing on the advantages of integrating quantitative and qualitative methods, the framework has the capacity to inform research that focuses on translating generated evidence into practice. Using the MMTR framework to guide a study design process allows translational science researchers to envision how the intersection of mixed methods with translational research components can be used to develop evidence-based, scientifically sound and participant-centered plans for improvement, thereby enhancing knowledge translation from inception to its adoption, spread and sustainability (Ivankova et al., 2018).

Figure 34.2  Mixed methods translational research process MMR: mixed methods research; TR: translational research

501

502

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Figure 34.3  Mixed methods translational research framework Reprinted from Ivankova, Herbey, and Roussel (2019) with permission of Dialectical Publishing.

The MMTR framework captures the six phases of the action research cycle informed by mixed methods research, which is in the centre of the framework. The cycle is further embedded in the processes of adaptation, dissemination and implementation that help translate the innovation informed by EBP. Because adaptation, dissemination and implementation are ongoing until the innovation is adopted and its sustainability is established in the community or practice, these processes can strategically influence the purpose and design of each phase in the community-based participatory action research cycle. The six phases—diagnosing, reconnaissance, planning, acting, evaluation, and monitoring— conceptually follow the cycle of the action research methodological steps outlined by Lewin (1948). The phases follow each other sequentially within a cycle while also interacting with each other in multiple ways depending on the purposes and needs of translational research tasks. During each phase, conceptual, procedural and inferential aspects of mixed methods are used to inform methodological and procedural decisions related

to the goals of each phase. For example, during the diagnosing phase, goals of knowledge translation and stage in the translational research continuum are identified along with the rationale for addressing the problem using a mixed methods approach. The need to integrate multiple sources of evidence provides a framework for developing research questions that rely on the integration of quantitative information with qualitative data generated with stakeholders’ input. Full-scale mixed methods studies consisting of conceptualization, implementation and development of integrated inferences (Tashakkori et  al., 2021) are conducted in the reconnaissance and evaluation phases. Using a mixed methods design during the reconnaissance phase, translational science researchers collect, analyze and integrate quantitative and qualitative data to assess community needs, potential acceptance of EBP-informed innovation, and the community resources necessary for disseminating and implementing an innovative program or intervention. Similarly, a mixed methods design is used to frame evaluation of the intervention or programme implementation with

REALIZING METHODOLOGICAL POTENTIALS AND ADVANTAGES

the purpose of informing further actions depending on the results from evaluation. For example, Younas et  al. (2019) used a sequential Qualitative à Quantitative design in the reconnaissance phase to identify challenges nurse educators experience when incorporating the World Health Organization’s (WHO) recommended competencies in nursing education in Pakistan. The researchers first conducted in-depth interviews with 12 nurse educators and used the results to develop a questionnaire exploring common challenges in clinical and academic teaching to inform changes in policies and practice. Alternatively, Ruiz-Íñiguez et al. (2021) employed a sequential Quantitative à Qualitative design in the evaluation phase to explore the acceptability and effectiveness of a culturally adapted mindfulness-based intervention for mental health professionals in Cuba. The researchers first collected and analyzed survey data from 30 Cuban participating psychologists to assess the effectiveness of the intervention and then followed up with individual interviews of the same professionals to better understand the intervention acceptability. In Wu et  al.’s (2020) study, the researchers implemented a concurrent Quantitative + Qualitative design in the evaluation phase to assess the effectiveness and practicability of an oral hygiene education program for long-term care workers at a Veterans General Hospital in Taiwan. The researchers integrated quantitative survey data assessing pre- and post-intervention oral hygiene knowledge and participants’ perspectives on the practicability of the programme obtained via semistructured interviews with 80 long-term care workers to inform programme implementation in the hospital practice. Breimaier et  al.’s (2015) study illustrates another application of the concurrent Quantitative + Qualitative design. The study aim was to identify effective strategies for the implementation of evidence-based guidelines by nurses in an Austrian teaching hospital for preventing falls among older patients. The researchers used a concurrent Quantitative + Qualitative design in both reconnaissance and evaluation phases to first explore nurses’ attitudes and potential facilitators and barriers to guideline implementation and then to evaluate the effectiveness of the tailored implementation strategies. The mixed methods inferences produced by integrating quantitative and qualitative data and results in the reconnaissance and evaluation phases, inform planning, implementation and monitoring of the intervention. For example, during the planning phase, inferences from the reconnaissance phase are discussed with the key stakeholders to inform the development of the intervention that will be responsive to their needs. During the acting phase,

503

the action is taken to start the process of innovation adaptation and implementation as informed by the mixed methods inferences from the preceding phases. If the programme or intervention has already been used in the community, action is directed to further refinements and adjustments of the intervention to facilitate its acceptability and adoption in the community. The monitoring phase relies on the mixed methods inferences from intervention evaluation, which help guide decisions about further adaptation and the D&I process necessary to reach desired outcomes. If the intervention or programme is successful, continuous mixed methods evaluation of its progress can promote further monitoring and sustainability, thereby enabling transferability of the evidence-based results to other contexts and community settings. Alternatively, if the desired outcomes are not achieved, a decision can be made either to revise the intervention using the results from a mixed methods evaluation or to return to the diagnosing and reconnaissance phases to better conceptualize the problem by conducting a more in-depth mixed methods exploration of the issue to inform the revisions. Such cyclical, dynamic and iterative nature of community-based participatory action research supports the translational research goals and increases chances that the innovation will be accepted and used in the community. We used the MMTR methodological framework to elicit contemporary perspectives on designing mixed methods studies for knowledge translation by interviewing researchers who apply mixed methods in their translational research practice. We share these perspectives in the following section.

DESIGNING MIXED METHODS STUDIES FOR KNOWLEDGE TRANSLATION: INSIGHTS AND PERSPECTIVES To obtain a multidisciplinary perspective on designing mixed methods studies for knowledge translation, we interviewed 35 translational scientists from two countries: US (n = 30) and Canada (n = 5) using Zoom web conferencing. Participants represented 10 disciplines (education, engineering, nutrition, medicine, psychology, public health, medical anthropology, nursing, health services and social work). The majority (n = 31) served as faculty at academic institutions, three represented non-profit research organizations and one was a programme officer from the National Institutes of Health. Two-thirds were in the senior rank of a full and associate professor (n = 22).

504

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

We used the MMTR methodological framework to inform the interview questions, which focused on the participants’ experiences with designing and conducting mixed methods studies across the knowledge translation continuum from T1 to T4 and how mixed methods can optimize translational research through intersecting with its synergistic components (EBP, adaptation, D&I, CBPR and action research). We also explored the participants’ perspectives on what researchers should be aware of when designing and implementing mixed methods studies for knowledge translation and the related challenges. We organized these perspectives in five themes that span different aspects of using mixed methods research to design and conduct studies for knowledge translation.

Mixed Methods Research Applies to All Stages of the Translational Research Continuum The study participants believed that mixed methods can be effectively applied, both conceptually and procedurally, in all stages of the knowledge translation continuum, from T1 to T4: “Mixed methods could be used in all aspects of the continuum.” At T1 stage (translation from basic science to humans), translational science researchers should think about the potential utility of their study results by considering the stakeholders’ perspectives and the quantitative indicators. As one participant observed, “Mixed methods needs to start from T1.” At T2 (translation of research findings to develop new treatments and programmes), evidence-based guidelines are developed by combining evidence from multiple quantitative and qualitative sources. Developing a meaningful intervention requires understanding of the intervention context, the implementation process and the population group that will be receiving the intervention. During T3 (translation of evidence-based guidelines into everyday practice) and T4 (translation to the community and real-world settings), employing mixed methods provides a holistic perspective by adding greater depth and scope to complex processes and allowing better understanding of intervention contexts and users by investigating their needs and values. This requires combining personal experiences of people with measurable quantitative outcomes. During T3 and T4 stages, the use of mixed methods can be more prominent due to an exploratory nature of these studies, focus on innovation application and the need to engage communities. One participant noted, “T3 and T4

… start dealing with people, and people are messy, and so mixed approaches to understanding how evidence is applicable becomes very, very great.”

Mixed Methods Research Provides the Context for Designing Studies for Knowledge Translation Understanding intervention context is critical in translational science. Mixed methods research provides tools to understand how an intervention is received and supported by different stakeholders in specific settings and cultures: “understanding context, which I think is really important in implementation science.” With emphasis on qualitative stakeholder-oriented methods, mixed methods research allows for creating contextual understandings of what may impact the process of knowledge translation by exploring stakeholders’ perspectives, needs and experiences and integrating them with the quantitative outcomes data. A participant stated: “it [mixed methods] provides the best means possible to understand humans in the entirety of their circumstances.” Another participant pointed out: “Mixed methods can give us clues about the sort of contextual or sort of environmental factors that may influence an intervention’s implementation. And so, if we know those factors, then we can adapt our intervention accordingly.” Relying on either quantitative or qualitative methods is not sufficient to understand the complex intervention setting and the scope of needed cultural adaptations. A participant stated: “You’re trying to get a sense of where your research is gonna be applied and getting a sense of the context of that setting is really important, and I think that it’s hard to get that with either method, quantitative or qualitative by itself.” Thus, using mixed methods serves to connect the dots in “making sense” and adding greater depth of meaning to research evidence and practice.

Mixed Methods Research Design Addresses the Process and Outcomes of Knowledge Translation The mixed methods approach helps connect researchers to the real world because it allows for the study of both the course and outcomes of issues using cyclical action-oriented and communitybased methods, thereby facilitating the process of knowledge translation and bridging the gap between research and practice. One study participant shared:

REALIZING METHODOLOGICAL POTENTIALS AND ADVANTAGES

One of the challenges I saw from very early in my career was that the translation of research to practice had a huge gap. Many researchers don’t know how to make that translation and practitioners don’t know how to do that translation. And so, I see translational research and mixed methods as a way to help bridge that gap. And to engage in a research to practice process.

Using mixed methods research design can help identify why certain implementation strategies are not successful and “why some interventions turn out to be more widely adopted than others”. Additionally, mixed methods can enhance dissemination of the EBP practices by exploring multiple stakeholders’ views about effective communication strategies: “in thinking about, trying to bring clinicians or policymakers or other kinds of decision-makers, trying to understand how to communicate results to them so that it can be better translated. I think mixed methods can be applicable there too in understanding, you know, what communication is gonna be most effective.”

Mixed Methods Research Design Helps Address Complex Knowledge Translation Questions Knowledge translation is a complex multi-stage and multilevel process. Mixed methods is viewed as a rigorous methodology that can address complex translational research problems by answering system level questions, focusing on practically relevant outcomes and considering users’ perspectives. For example, one participant observed with relevance to mixed methods research design: “It is creating the evidence base, certain types of evidence resonate with certain audiences.” Another participant noted that mixed methods “provides the best means possible to understand humans in the entirety of their circumstances”. These statements underscore the ability of mixed methods to provide enhanced understanding of the perspectives of the population being examined, resulting in a more complete understanding of complex phenomena. Moreover, participants noted the creative aspect of mixed methods designs that suit the purpose of translational research well: “I am more and more open to creative study designs that are good for real world use.” Integrating multiple participants’ perspectives with supporting quantitative evidence and a high attention to the quality of the research design results in more rigorous study outcomes to inform adaptation, dissemination and implementation of the innovation. A participant stated that “when

505

both sets of methods are utilized, it gives the audience a greater confidence in the validity of the findings.” Whereas another participant highlighted the relevance of mixed methods for knowledge translation in the following way: “a desire to make some changes … it forces you to get beyond a single method, a single perspective.”

Mixed Methods Research Design Considerations for Knowledge Translation Translational science researchers also shared thoughts about the utility of mixed methods designs for knowledge translation purposes based on their practice of designing mixed methods studies across the knowledge translation continuum. Participants noted the advantages of using sequential designs as fitting the purposes of intervention adaption, dissemination and implementation. For example, using a sequential Quantitative à Qualitative design is advantageous for assessing the intervention outcomes and for its further dissemination and implementation: “a sequential design, at the end, you can come back with a qualitative component that really allows you to go back and explore what happened or re-examine whatever conclusions you’re making about impact and about dissemination by asking sort of a focused inquiry with your qualitative methods.” Qualitative follow-up is particularly important for obtaining stakeholders’ perspectives on the feasibility and efficacy of the intervention. A study participant stated: And really without a qualitative component especially a stakeholder informed component, it’s often hard to understand why translation was or wasn’t successful. Why it was successful for certain groups or situations and not for others? So, that really in … implementation and dissemination science … the qualitative approach can really inform our understanding of what we see quantitatively.

A sequential Qualitative à Quantitative design was noted to be useful for designing new measures to inform intervention sustainability and for developing implementation models and frameworks. One participant explained: I used the mixed method approach in developing a way of measuring the use of research evidence in policy and practice. I’ve used it to develop a measure of the kinds of cultural exchanges that occur among different stakeholders in translational research or translational projects, I should say …

506

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

and I used it more recently … to develop a way of measuring sustainability of prevention programs and initiatives.

Beginning with the qualitative component is also recommended when there is a need for cultural adaptions and understanding individual and system factors that shape the knowledge translation process: “prospectively it can help guide tailoring and adaptation of both the intervention and the approaches for implementing that intervention.” A concurrent Qualitative + Quantitative design was perceived useful in situations requiring timely assessment of the intervention process, as explained by a study participant: “the more of those [mixed methods designs] can be concurrent that’s usually helpful for us because we want to know … what’s happening at specific time points in a project, in a process.” One participant noted the value of combining mixed methods with case study methodology as an effective approach for knowledge translation: “anytime you wanna get into questions of why is a person … you know, where did the barriers … where do things break down? Where do … if things worked really well, right? You know, … I had case study methodologies where I found like perfect examples of translation.” In summary, these perspectives of translational science researchers who use mixed methods in their research provide insight into how mixed methods research can be used in designing studies for knowledge translation. Mixed methods research design holds an important role in supporting translational research and provides a methodological framework for integrating multiple components of the translational research process to effectively achieve translational goals.

ADDITIONAL CONSIDERATIONS AND FUTURE DEVELOPMENTS Intersection of mixed methods with translational research, while advantageous, may cause certain challenges to researchers, particularly those who are new to mixed methods and translational research. Designing mixed methods studies for knowledge translation requires considerations that go beyond general requirements for expertise in quantitative, qualitative and mixed methods research and knowledge of translational research methods (Fudge et  al., 2016; Halcomb, 2019; Wooten et al., 2014). As we have shown, knowledge translation is not a straightforward process

and involves interaction of the many methodological components that have established conceptual and procedural characteristics at the epistemological and design levels. Considering these characteristics is important for identifying the focus for a MMTR study (e.g., intervention adaptation to the needs of a certain group or community; or disseminating evidence-based positive outcomes from an intervention for further monitoring) and aligning this focus with the methodological features of a mixed methods design (e.g., sequence and integration of qualitative and quantitative components). Additional project-related challenges involve lack of or limited experience in collecting, management and analysis of large volumes of data; sufficient time to complete all phases of the MMTR project; and having access to resources and facilities to implement the mixed methods research design. Mixed methods researchers should also consider personal, interpersonal and organizational factors that have implications for designing effective mixed methods studies for knowledge translation. At a personal level, researchers may be challenged by the need to develop mixed methods translational research thinker philosophy, which requires investigators to have a set of beliefs and behaviours to support and promote the intersection of mixed methods with translational research components. Becoming a mixed methods translational research thinker involves thinking in the context of the intervention to recognize the setting and potential value for stakeholders, focusing on the process of knowledge translation as well as the outcomes, showing interest in streamlining the timeframe from innovation development to its adoption in practice and community, being willing to let others lead with their area of expertise, and being open to exploring alternative methods, such as mixed methods to design studies for knowledge translation. As one of the interviewed translational researchers stated: “a desire to make some changes … it forces you to get beyond a single method, a single perspective.” Identifying collaborators with personal characteristics that support the intersection of mixed methods with translational research, establishing discipline-specific team leaders, and developing culturally responsive teams may be challenging too (Fudge et al., 2016; Roberge-Dao et al., 2019; see also Chapter 16, this volume). For example, Roberge-Dao et  al. (2019) used a sequential Qualitative à Quantitative mixed methods design to develop an integrated knowledge translation model for rehabilitation practice. The authors reported considerable challenges in developing a system of shared leadership that clearly indicated the role of each team member, enabled

REALIZING METHODOLOGICAL POTENTIALS AND ADVANTAGES

their meaningful contributions to the translational endeavor, and ensured that the goals of the collaboration were met. One way to address this challenge is to adopt a distributed leadership approach that enables research team leaders with discipline-specific expertise to retain tactical decision-making, while allowing an executive leadership committee consisting of stakeholders and researchers to manage the overarching strategic vision (Anderson et al., 2021). Developing collaborative partnerships with the key stakeholders, considering community priorities, and engaging community members with research are key factors in designing mixed methods studies for knowledge translation (Ivankova & Johnson, 2022). However, communicating with community partners about the mixed methods design and achieving consensus about the MMTR study process is challenging and requires building their mixed methods and translational research capacity and developing their sense of ownership for the innovation. Additionally, establishing common goals for knowledge translation and developing inter-organization and multidisciplinary collaborative teams can be effective for leveraging mixed methods and translational research expertise, particularly if such expertise is not fully available within a single institution or organization (Wooten et al., 2014). In conclusion, intersecting mixed methods with translational research and its integral components, such as EBP, adaptation, D&I, CBPR and action research, offers methodological advantages for designing studies to optimize knowledge translation. The flexibility of mixed methods research to intersect with these translational research components and its ability to facilitate different stages in the translational research continuum make it advantageous for addressing complex knowledge translation problems. While MMTR studies remain prevalent in health sciences research, their use across disciplines is expanding. With the growing demand for facilitating the implementation of evidence-based practices in professional and community settings, more translational science researchers consider using mixed methods due to its ability to provide contextual understandings of the knowledge translation process and to explore knowledge translation problems at different levels and from different perspectives. This trend is promising and holds a lot of potential for innovative applications of mixed methods designs in knowledge translation studies shaped by unique disciplinary contexts and knowledge translation goals. Besides complying with the methodological characteristics of the translational research process, as discussed in this chapter, these design applications should be guided by

507

unique disciplinary conventions, accepted epistemological practices, and established criteria for evidence quality in that discipline. Adopting a mixed methods translational research thinker philosophy may help researchers who embark on this journey to foresee these challenges and design effective mixed methods studies for knowledge translation.

ACKNOWLEDGEMENT This work was supported by 2018–19 The University of Alabama at Birmingham Faculty Development Grant Program (FDGP).

WHAT TO READ NEXT Green, C. A., Duan, N., Gibbons, R. D., Hoagwood, K. E., Palinkas, L. A., & Wisdom, J. P. (2015). Approaches to mixed methods dissemination and implementation research: Methods, strengths, caveats, and opportunities. Administration and Policy in Mental Health and Mental Health Services Research, 42, 508–523. doi:10.1007/s10488-014-0552-6

This paper describes the use of mixed methods in dissemination and implementation research and discusses the advantages of combining multiple quantitative and qualitative methods within mixed methods designs. Ivankova, N.V., Anderson, J.L., Herbey, I. I., Roussel, L., & Kim, D. (2023). Intersecting mixed methods with translational research: Implications for educational research and practice. In: Tierney, R.J., Rizvi, F., & Erkican, K. (Eds.). International Encyclopedia of Education, 12 (pp. 588-598). Elsevier. https:// dx.doi.org/10.1016/B978-0-12-818630-5. 11058-9

This chapter provides a detailed discussion of the advantages and challenges of intersecting mixed methods with translational research and explains how mixed methods combined with evidencebased practice, adaptation, dissemination and implementation, community-based participatory research, and action research facilitates the knowledge translation process. Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration and Policy in Mental Health, 38(1), 44–53. doi:10.1007/s10488-010-0314-z

508

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

This paper describes the use of mixed methods designs in implementation science research guided by Aarons et al.’s (2010) conceptual framework of the implementation process consisting of four phases: exploration, adoption/preparation, implementation and sustainment. The paper focuses on the advantages and methodological characteristics of integrating different methods to address the purposes of each implementation phase.

REFERENCES Allen, J. D., Linnan, L. A., & Emmons, K. M. (2012). Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. In R. C. Brownson, G. A. Colditz, & Proctor, E. K. (Eds.). Dissemination and implementation research in health: Translating science to practice (pp. 281– 304). Oxford University Press. Anderson, J. L., Reamey, R. A., Levitan, E. B., M Asif, I., S Aswani, M., Fletcher, F. E., … & Mugavero, M. J. (2021). The University of Alabama at Birmingham COVID-19 Collaborative Outcomes Research Enterprise: Developing an institutional learning health system in response to the global pandemic. Learning Health Systems, e10292. https://doi. org/10.1002/lrh2.10292 Baydala, L., Fletcher, F., Worrell, S., Kajner, T., Letendre, S., Letendre, L., & Rasmussen, C. (2014). Partnership, knowledge translation, and substance abuse prevention with a First Nations Community. Progress in Community Health Partnerships: Research, Education, and Action, 8, 145–155. https://doi.org/10.1353/cpr.2014.0030 Breimaier, H. E., Halfens, R. J., & Lohrmann, C. (2015). Effectiveness of multifaceted and tailored strategies to implement a fall-prevention guideline into acute care nursing practice: A before-andafter, mixed-method study using a participatory action research approach. BMC Nursing, 14(1), 18. https://doi.org/10.1186/s12912-015-0064-z Callard, F., Rose, D., & Wykes, T. (2011). Close to the bench as well as the bedside: Involving service users in all phases of translational research. Health Expectations, 15, 389-400. https://doi. org/10.1111/j.1369-7625.2011.00681.x Canadian Institutes of Health Research (CIHR) (2016). Knowledge translation. Retrieved from: www.cihrirsc.gc.ca/e/29418.html Chambers, D. A., & Norton, W. E. (2016). The adaptome: Advancing the science of intervention adaptation. American Journal of Preventive Medicine, 51(4), S124–S131. https://doi.org/10.1016/j. amepre.2016.05.011

Collins, F. S. (2011). Reengineering translational science: The time is right. Science of Translational Medicine, 3(90), 1-6. https://doi.org/10.1126/ scitranslmed.3002747 Craig, S. L. (2011). Precarious partnerships: Designing a community needs assessment to develop a system of care for gay, lesbian, bisexual, transgender and questioning (GLBTQ) youths. Journal of Community Practice, 19, 274–291. https://doi.org /10.1080/10705422.2011.595301 Creswell, J. W., Klassen, A. C., Plano Clark, V. L., & Smith, K. C. for the Office of Behavioral and Social Sciences Research. (2018). Best practices for mixed methods research in the health sciences (2nd ed.). Bethesda, MD: National Institutes of Health. Retrieved from https://obssr.od.nih.gov/wp-con tent/uploads/2018/01/Best-Practices-for-MixedMethods-Research-in-the-Health-Sciences-201801-25.pdf Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Crime and Justice Institute at Community Resources for Justice (2009). Implementing evidence-based policy and practice in community corrections (2nd ed.). Washington, DC, National Institute of Corrections. Curry, L., & Nunez-Smith, M. (2015). Mixed methods in health sciences research: A practical primer. Sage. Darwichea, H., Barnes, M., Barnes, L., Cooper, L., Bokor, J., & Koroly, M. (2017). Bench to bedside: The effectiveness of a professional development program focused on biomedical sciences and action research. Science Education (Arlington), 26(1), 32–47. DeJonckheere, M., Lindquist-Grantz, R., Toraman, S., Haddad, K. and Vaughn, L. M. (2019). Intersection of mixed methods and community-based participatory research: A methodological review. Journal of Mixed Methods Research, 13(4), 481–502. https://doi.org/10.1177%2F1558689818778469 de Visser, R. O., Graber, R., Abraham, C., Hart, A., & Memon, A. (2020). Resilience-based alcohol education: Developing an intervention, evaluating feasibility and barriers to implementation using mixed-methods. Health Education Research, 35(2), 123–133. https://doi.org/10.1093/her/cyaa006 Faridi, Z., Grunbaum, J.A., Gray, B.S., Franks, A., & Simoes, E. (2007). Community-based participatory research: Necessary next steps. Preventing Chronic Disease, 4(3). Retrieved from: www.cdc.gov/pcd/ issues/2007/jul/06_0182.htm Fort, D. G., Herr, T. M., Shaw, P. L., Gutzman, K. E., & Starren, J. B. (2017). Mapping the evolving definitions of translational research. Journal of Clinical and Translational Science, 1(1), 60–66. doi:10.1017/cts.2016.10

REALIZING METHODOLOGICAL POTENTIALS AND ADVANTAGES

Fudge, N., Sadler, E., Fisher, H. R., Maher, J., Wolfe, C. D., & McKevitt, C. (2016). Optimizing translational research opportunities: A systematic review and narrative synthesis of basic and clinician scientists’ perspectives of factors which enable or hinder translational research. PLoS One, 11(8), e0160475. https://doi.org/10.1371/journal. pone.0160475 Green, C. A., Duan, N., Gibbons, R. D., Hoagwood, K. E., Palinkas, L. A., & Wisdom, J. P. (2015). Approaches to mixed methods dissemination and implementation research: Methods, strengths, caveats, and opportunities. Administration and Policy in Mental Health and Mental Health Services Research, 42, 508–523. https://doi.org/10.1007/ s10488-014-0552-6 Green, L. W., & Mercer, S. L. (2001). Can public health researchers and agencies reconcile the push from funding bodies and the pull from communities? American Journal of Public Health, 91(12), 1926– 1929. doi:10.2105/AJPH.91.12.1926 Greene, J. C. (2007). Mixed methods in social inquiry. Jossey-Bass. Hacker, K. (2013). Community-based participatory research. Sage. Halcomb, E. J. (2019). Mixed methods research: The issues beyond combining methods. Journal of Advanced Nursing, 75, 499–501. http://dx.doi. org/10.1111/jan.13877 Hull, L., Goulding, L., Khadjesari, Z., Davis, R., Healey, A., Bakolis, I., & Sevdalis, N. (2019). Designing high-quality implementation research: Development, application, feasibility and preliminary evaluation of the implementation science research development (ImpRes) tool and guide. Implementation Science, 14(1), 1–20. https://doi. org/10.1186/s13012-019-0897-z Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (2001). Community-based participatory research: Policy recommendations for promoting a partnership approach in health research. Education for Health, 14, 182–197. doi:10.1080/ 13576280110051055 Ivankova, N. V. (2015). Mixed methods applications in action research: From methods to community action. Sage. Ivankova, N. V. (2017). Applying mixed methods in community-based action research: A framework for engaging stakeholders with research as a means for promoting patient-centeredness. ­Journal of Research in Nursing, 22(4), 282-294. doi: 10.1177/1744987117699655 Ivankova, N.V., Anderson, J.L., Herbey, I. I., Roussel, L., & Kim, D. (2023). Intersecting mixed methods with translational research: Implications for educational research and practice. In: Tierney, R.J., Rizvi, F., & Erkican, K. (Eds.). International Encyclopedia of Education, 12 (pp. 588-598). Elsevier.

509

https://dx.doi.org/10.1016/B978-0-12-8186305.11058-9 Ivankova, N. V., Herbey, I. & Roussel, L. (2018). Theory and practice of using mixed methods in translational research: A cross-disciplinary perspective. International Journal of Multiple Research Approaches, 10(1), 356–372. https://doi. org/10.29034/ijmra.v10n1a24 Ivankova, N. V. & Johnson, S. L. (2022). Designing integrated mixed methods action research studies. In J. Hitchcock & A. Onwuegbuzie (Eds.), Routledge Handbook for Advancing Integration in Mixed Methods Research (pp. 158–173). Routledge. Ivankova, N. V., & Wingo, N. (2018). Applying mixed methods in action research: Methodological potentials and advantages. The American Behavioral Scientist, 62(7), 978–997. https://doi.org/10.11 77%2F0002764218772673 Ivankova, N.V., & Wingo, N.P. (2021). Intersecting mixed methods and community-based participatory action research to promote patient-centeredness in research. In P. Liamputtong (Ed.) Handbook of Social Inclusion, Research and Practices in Health and Social Sciences. Springer. https://doi. org/10.1007/978-3-030-48277-0_90-1 Ivankova, N. V. and Wingo, N. (2023). Intersecting mixed methods with action and community-based participatory research approaches. In: Tierney, R.J., Rizvi, F., & Erkican, K. (Eds.). International Encyclopedia of Education, 12 (pp. 567-580). Elsevier. https://dx.doi.org/10.1016/B978-0-12-8186305.11056-5 Johnson, R. B. (2017). Dialectical pluralism: A metaparadigm whose time has come. Journal of Mixed Methods Research, 11, 156–173. https://doi.org/ 10.1177%2F1558689815607692 Johnson, S., & Ivankova, N.V. (2019). Intersecting mixed methods and action research to evaluate a community-based care coordination service: Implications for designing stakeholder-informed and context-specific health services research studies. International Journal of Multiple Research Approaches, 11(1). https://ijmra.org/volume11-number-1-april-2019/ Kajermo, K. N., Boström, A. M., Thompson, D. S., Hutchinson, A. M., Estabrooks, C. A., & Wallin, L. (2010). The BARRIERS scale – the barriers to research utilization scale: A systematic review. Implementation Science, 5(1), 32. https://doi. org/10.1186/1748-5908-5-32 Khoury, M. J., Gwinn, M., & Ioannidis, J. P. (2010). The emergence of translational epidemiology: From scientific discovery to population health impact. American Journal of Epidemiology, 172(5), 517–524. https://doi.org/10.1093/aje/kwq211 Khoury, M. J., Gwinn, M., Yoon, P. W., Dowling, N., Moore, C. A., & Bradley, L. (2007). The continuum

510

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

of translation research in genomic medicine: How can we accelerate the appropriate integration of human genome discoveries into health care and disease prevention? Genetics in Medicine, 9(10), 665–674. https://doi.org/10.1097/GIM. 0b013e31815699d0 Kitson, A., Harvey, G., & McCormack, B. (1998). Enabling the implementation of evidence based practice: A conceptual framework. Quality in Health Care, 7, 149–158. http://dx.doi.org/ 10.1136/qshc.7.3.149 Knoepke, C.E., Ingle, M.P., Matlock, D.D., Brownson, R.C., & Glasgow, R.E., (2019). Dissemination and stakeholder engagement practices among dissemination and implementation scientists: Results from an online survey. PLoS One 14, e0216971. Layde, P., Christiansen, A., Peterson, D., Guse, C., Maurana, C., & Brandenburg, T. (2012). A model to translate evidence-based interventions into community practice. American Journal of Public Health, 102, 617–624. doi:10.2105/AJPH.2011. 300468 Lewin, G. (1948). (Ed.). Resolving social conflicts: Selected papers on group dynamics by Kurt Lewin. Harper & Brothers. Lucero, J., Wallerstein, N., Duran, B., Alegria, M., Greene-Moton, E., Israel, B., … and Schulz, A. (2018). Development of a mixed methods investigation of process and outcomes of communitybased participatory research. Journal of Mixed Methods Research, 12(1), 55–74. https://doi.org/ 10.1177%2F1558689816633309 Maloney, S., Tunnecliff, J., Morgan, P., Gaida, J. E., Clearihan, L., Sadasivan, S., … & Reynolds, J. (2015). Translating evidence into practice via social media: A mixed-methods study. Journal of Medical Internet Research, 17(10), e242. doi:10.2196/ jmir.4763 Martin, R. (2007). How successful leaders think. Harvard Business Review, 85(6), 60–67. MacDermid, J. C., & Graham, I. D. (2009). Knowledge translation: Putting the “practice” in evidencebased practice. Hand Clinics, 25(1), 125–143. https://doi.org/10.1016/j.hcl.2008.10.003 McKleroy, V. S., Galbraith, J. S., Cummings, B., Jones, P., Harshbarger, C., Collins, C.,… & ADAPT Team. (2006). Adapting evidence–based behavioral interventions for new settings and target populations. AIDS Education & Prevention, 18(supp.), 59–73. doi:10.1521/aeap.2006.18.supp.59 McNiff, J., & Whitehead, J. (2011). All you need to know about action research (2nd ed.). Sage. Meister, L. (2018). On methodology: How mixed methods research can contribute to translation studies. Translation Studies, 11(1), 66–83. https:// doi.org/10.1080/14781700.2017.1374206 Melnyk, B. M., & Fineout-Overholt, E. (2019). Evidence-based practice in nursing & healthcare: A

guide to best practice. (4th ed.). Lippincott Williams & Wilkins. National Institutes of Health (2021). Dissemination and implementation research in health (R01, Clinical Trial Optional). PAR-19-274. Naylor, M. D. (2018). Overview and summary: Translational research: From knowledge to practice. The Online Journal of Issues in Nursing, 23(2), OS. https://doi.org/10.3912/OJIN.Vol23No02ManOS Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration and Policy in Mental Health, 38(1), 44–53. https://doi.org/10.1007/s10488-0100314-z Palinkas, L. A., & Cooper, B. R. (2017). Mixed methods evaluation in dissemination and implementation science. In R. C. Brownson, G. A. Colditz, & E. K. Proctor, (Eds.). Dissemination and implementation research in health: Translating science to practice (2nd ed., 335–353). Oxford University Press. Plano Clark, V., & Ivankova, N. (2016). Mixed methods research: A guide to the field. Sage. Rabin, B. A., Brownson, R. C., Haire-Joshu, D., Kreuter, M. W., & Weaver, N. L. (2008). A glossary for dissemination and implementation research in health. Journal of Public Health Management and Practice, 14(2), 117–123. doi: 10.1097/01. PHH.0000311888.06252.bb Roberge-Dao, J., Yardley, B., Menon, A., Halle, M. C., Maman, J., Ahmed, S., & Thomas, A. (2019). A mixed-methods approach to understanding partnership experiences and outcomes of projects from an integrated knowledge translation funding model in rehabilitation. BMC Health Services Research, 19(1), 1–16. https://doi.org/10.1186/ s12913-019-4061-x Rogers, E. M. (1995). Diffusion of innovations (4th ed.). The Free Press. Roux, P., Donadille, C., Magen, C., Schatz, E., Stranz, R., Curado, A., … & Eurosider study group. (2021). Implementation and evaluation of an educational intervention for safer injection in people who inject drugs in Europe: A multi-country mixed-methods study. International Journal of Drug Policy, 87, 102992. https://doi.org/10.1016/j. drugpo.2020.102992 Ruiz-Íñiguez, R., Pérez-Díaz, R., García-Jacomino, J. C., Carralero-Montero, A., & Santed, M. Á. (2021). Acceptability and effectiveness of a mindfulnessbased intervention for reducing stress and burnout among mental health professionals: A mixedmethods pilot study conducted in Cuba. Current Psychology, 1–12. https://doi.org/10.1007/ s12144-021-01388-1 Shorten, A., & Smith, J. (2017). Mixed methods research: Expanding the evidence base.

REALIZING METHODOLOGICAL POTENTIALS AND ADVANTAGES

Evidence-Based Nursing, 20, 74–75. http://dx.doi. org/10.1136/eb-2017-102699 Straus, S. E., Tetroe, J., & Graham, I. (2009). Defining knowledge translation. Canadian Medical Association Journal, 181(3–4), 165–168. https://doi. org/10.1503/cmaj.081229 Stringer, E. T. & Aragón, A. O. (2020). Action research (5th ed.). Sage. Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012). Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine, 43, 337–350. https://doi.org/10.1016/j. amepre.2012.05.024 Tashakkori, A., Johnson, R. B. & Teddlie, C. (2021). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences (2nd ed.). Sage. Taylor, M. V., Priefer, B. A., & Alt-White, A. C. (2016). Evidence-based practice: Embracing integration. Nursing Outlook, 64, 575–582. https://doi. org/10.1016/j.outlook.2016.04.004 Wethington, E., & Burgio, L. D. (2015). Translational research on caregiving: Missing links in the translation process. In J. E. Gaugler & R. L. Kane (Eds.), Family caregiving in the new normal (pp. 193– 210). Academic Press.

511

Wooten, K. C., Rose, R. M., Ostir, G. V., Calhoun, W. J., Ameredes, B. T., & Brasier, A. R. (2014). Assessing and evaluating multidisciplinary translational teams: A mixed methods approach. Evaluation & the Health Professions, 37(1), 33–49. https://doi. org/10.1177%2F0163278713504433 Wu, S.-J., Wang, C.-C., Kuo, S.-C., Shieh, S.-H., & Hwu, Y.-J. (2020). Evaluation of an oral hygiene education program for staff providing long-term care services: A mixed methods study. International Journal of Environmental Research and Public Health, 17(12), 4429. https://doi. org/10.3390/ijerph17124429 Younas, A., Zeb, H., Aziz, S. B., Sana, S., Albert, J. S., Khan, I. U., … & Rasheed, S. P. (2019). Perceived challenges of nurse educators while teaching undergraduate nursing students in Pakistan: An exploratory mixed-methods study. Nurse Education Today, 81, 39–48. https://doi.org/10.1016/j. nedt.2019.07.002 Zoellner, J. M. & Porter, K. J. (2017). Translational research: Concepts and methods in dissemination and implementation research. In: A. M. Coulston, C. J. Boushey, M. G. Ferruzzi, & L. M. Delahanty (Eds.), Nutrition in the Prevention and Treatment of Disease (4th ed.) (pp.125–143). Academic Press.

35 Opportunities and Challenges for a Transdisciplinary Mixed Methods Research Future Mandy M. Archibald

INTRODUCTION The complexities inherent in understanding and addressing persistent and “wicked” social problems have catalyzed an interest in transdisciplinary and mixed methods research approaches. While transdisciplinary research and mixed methods research continue evolving as respective methodological orientations, their intersection remains underexplored. Yet, transdisciplinary research and mixed methods research emphasize integration as a scientific principle (Bergmann et  al., 2012) and are positioned as answers to complexity in their own right (Archibald, 2016; Poth, 2018). As such, their merger presents opportunities for an enhanced integrative logic with potentials for real-world impacts. Indeed, transdisciplinary research and mixed methods research share a number of underpinnings that when considered in tandem, offer unique opportunities and considerations pertinent to their synergistic combination. Simply stated, transdisciplinary research is research that supersedes the boundaries of any one discipline (Klein, 2018). It emphasizes integration, which Bergmann and colleagues (2012) regard as a push-back against the progressive differentiation of science evident throughout the

19th century. Similarly, the benefits of mixed methods research are often described according to the unique offerings made possible by the bringing together of methodologies or research methods, which result in newer, deeper or more explanatory perspectives on phenomena (Greene et al., 1989; Johnson et al., 2017). While integration can focus on myriad components of the research process including sampling (Collins et  al., 2007; Teddlie & Yu, 2007), data sources (Bazeley, 2012), and methods (Archibald & Gerber, 2018; Creamer & Schoonenboom, 2018) for example, integration can also involve the intentional bringing together of investigators from different disciplines (e.g., investigator triangulation) (Archibald, 2016). This latter point, the humanistic component of integration, has been less of a focus in the mixed methods literature as compared with the integration of data sources and research approaches more generally (Hesse-Biber, 2016). Yet, in a research and social climate increasingly focused on collaborative research, the human components of integration are integral to the success of a mixed methods research endeavour. Similarly, discourse in mixed methods research literature has often been occupied by discussions of narrative (qualitative) and statistical (quantitative) data. Participatory mixed methods, which emphasize involvement of community and stakeholder

OPPORTUNITIES AND CHALLENGES FOR A TRANSDISCIPLINARY FUTURE

partners throughout the research (DeJonckheere et  al., 2018) and the transformative paradigm (Mertens, 2010), which emphasizes social justice, mirror dialogue in the transdisciplinary research literature wherein a necessary component of research design is the inclusion of community actors (Bergmann et al., 2012). Regardless of the nuances of the transdisciplinary inclusions, epistemic boundaries crossed and disciplines involved, the human dimension of transdisciplinary research is a common thread superseding the multiple iterations, conceptualizations and applications of transdisciplinary research. Simply stated, it is the bringing together of people, in all their complexities, representations and socialized identities, that permits the collaborative foundation for transdisciplinary mixed methods research. For additional discussions of community-based mixed methods research, see Chapter 29 (this volume), and the use of a transformative lens in mixed methods, see Chapter 4 (this volume). In response to this essential, human underpinning of transdisciplinary mixed methods research, this chapter explores the human component of integration—namely, the considerations for the bringing together of people for transdisciplinary mixed methods research. Drawing from the work of Bergmann and colleagues (2012) regarding methods for transdisciplinary research and teambased mixed methods research, I introduce parallels from the mixed and transdisciplinary bodies of literature, identifying key components of transdisciplinary mixed methods research, drawing influence from Greene’s 2008 framework for social science research methodology. This framework includes the four domains of philosophical assumptions and stances, inquiry logic (or methodology), guidelines for practice (the “how to” or practical advice), and sociopolitical commitments. The human and sociocultural aspects of research practices, later proposed by Lunde et al. (2013) as a fifth category, are also considered. The relevance of these domains is woven throughout the chapter and, while all are considered, I place particular emphasis on philosophical assumptions and stances, as well as the human and sociocultural aspects of research practice in relation to transdisciplinary mixed methods. In doing so, I first introduce paradigms and epistemic culture as foundations for transdisciplinary mixed methods research, emphasizing pragmatism and reflexive questioning. I then discuss collaborative approaches and integration, distinguishing transdisciplinary mixed methods research while incorporating the aforementioned philosophical considerations, drawing upon mixed methods literature on disciplinary mergers. I then provide practical guidance on the inner workings of

513

transdisciplinary mixed methods where dimensions of integration in transdisciplinary mixed method design and the importance of integrative conceptual work are discussed and practical strategies for teams are offered. I conclude with possibilities for a transdisciplinary mixed methods future, wherein the contributions of realist principles and tailored facilitation, communication and technological considerations, innovative mergers and recursive designs are emphasized.

PARADIGMS AND EPISTEMIC CULTURES: FOUNDATIONS FOR TRANSDISCIPLINARY MIXED METHODS RESEARCH Transdisciplinary research—like mixed methods research—has been defined and understood in myriad ways (Fam et  al., 2018; Johnson et  al., 2007). Each conceptualization carries unique research implications. For instance, definitions of transdisciplinary research emphasizing the bringing together of academic disciplines to surpass the contribution of each distinct discipline presuppose some level of foundational understanding—of scholarship, academic outputs and of the nature of academic work more generally. However, depending on the discipline, transdisciplinary research is also understood as the integration of “scientific and societal bodies of knowledge” (Popa et  al., 2015, p. 45), which requires expertise from within and beyond the scientific arena. Indeed, scholars such as Jahn and colleagues (2012) and Bergmann and colleagues (2012) position transdisciplinary research as a highly reflective and critical practice that, by merging social and scientific expertise, contributes to both social and scientific progress. In alignment with such an understanding, transdisciplinary research is well framed as a process, the success of which is contingent on the capabilities of human actors to exercise critique and reflexivity for insight. It follows that a transdisciplinary research future requires pronounced attention to the means of supporting this process (e.g., personnel, facilitation). Among the avenues to support and facilitate this process is how to achieve understanding of one’s own positionality, and the “importance of reflexive questioning of values, backgrounds assumptions, and normative orientations” (Popa et  al., 2015, p. 46), which Popa and colleagues argue, has not be sufficiently acknowledged. Recognizing then, the importance of reflexive questioning in the meaningful bringing together of diverse disciplines and social actors, any discussion of transdisciplinary research in accordance with mixed methods research requires dual

514

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

attention to research paradigms. Indeed, paradigms have historically occupied a central role in the mixed methods discourse, in relation to the integration of research methodologies, methods, as well as investigators (i.e., investigator triangulation; Archibald, 2016). As such, transdisciplinary mixed methods research requires sensitivity to paradigms, as these carry implications for research design as well as the functioning (process) and outcomes of the research endeavour.

Paradigms and Stances for Transdisciplinary Mixed Methods Research Philosophical assumptions and stances are one of four components of the framework for social science methodology identified by Greene in her framework, alongside inquiry logics, guidelines for practice and sociopolitical commitments (Greene, 2008). Philosophical assumptions guiding mixed methods and transdisciplinary research—as well as scientific inquiry in general—comprise ontological (reality/social world), epistemological (knowledge) and methodological components. Embedded within these stances are the roles of values (axiology) and context, and the presence and positioning of objectivity and subjectivity within research investigation(s). Mixing within mixed methods research has largely centred upon the philosophical and paradigmatic, as well as the inquiry logic—or methodology—domains. This discussion proceeds with a recognition that within mixed methods research, “ontological, epistemological, methodological, practical, collegial, and personal issues are to some extent entangled into each other” (Lunde et al., 2013, p. 198). While paradigms have been the catalyst for “wars” within the mixed methods research community, their perceived incompatibility can be traced back to how paradigms are conceptualized to begin with. Morgan (2007) identified four versions of paradigms that impact the mixing of methods, including paradigms such as: (1) worldviews (i.e., all encompassing ways of being and seeing the world); (2) epistemological stances (i.e., distinct belief systems); (3) shared beliefs among members of a specialty area (i.e., within a community which shares consensus about priority problems and corresponding methods); and (4) model examples (i.e., exemplars for how research should be done). Within these, operating from a perspective of paradigms as encompassing worldviews in particular has been linked to paradigmatic incommensurability: radically different ontological and epistemological assumptions create chasms

between paradigms, thwarting combination and communication between them. Rather, Morgan (2007) purports advantages to considering the four stances on paradigms as “nested within each other” (p. 54). In relation to transdisciplinary mixed methods, the concept of nested paradigmatic stances holds particular relevance. Specifically, individual epistemological stances may form the basis for shared beliefs within a specialty area, related in part to the socialization and shared culture experienced alongside members of one’s tribe—their epistemic culture (i.e., the socialization and locality of disciplinary knowledge) (Knorr-Cetina, 1999; Lunde et  al., 2013). Knowledge claims are constructed within communities through social, discursive and material practices, largely mediated by the organizational structure that gives way to their construction (Knorr-Cetina, 1999). Critically, it is the reliance on particular belief systems that “justifies the pursuit of different research questions and the use of different methods to answer those questions” (Morgan, 2007, p. 59). Given that transdisciplinary research involves the bringing together of investigators who operate within different socialized disciplines and, likely, belief systems associated with those disciplines (Archibald, 2016), then the combination (integration) of different belief systems vis-à-vis transdisciplinary research can give way to new problem conceptualizations, research questions and associated methods. However, it is a collective rather than an individual epistemic identity that facilitates such a collaborative orientation, thereby opening up such new possibilities (Knorr-Cetina, 1999). Yet, which paradigm(s) are well suited to facilitating such collaborative efforts and which techniques could facilitate such a combination of belief systems, requires further attention. For a discussion of the role of methodological paradigms for dialogic knowledge production, see also Chapter 6 (this volume). Comprehensive discussions of paradigms in mixed methods research and their applications can be found in the literature (e.g., Morgan, 2007; Shannon-Baker, 2016). Similarly, various stances towards paradigms can be observed, such as a dialectic paradigm stance (where the integrity of paradigms is preserved while recognizing their social construction), an alternative or single paradigm stance (where a new paradigm emerges), substance theory stance (exploring multiple theories), a-paradigmatic stance (where the focus is on technical rather than philosophical challenges), multiple paradigm stance (where various paradigms are occupied and debated) and the complementary strengths stance (where clear distinctions are drawn between paradigms) (Greene, 2007). Regarding transdisciplinary

OPPORTUNITIES AND CHALLENGES FOR A TRANSDISCIPLINARY FUTURE

mixed methods, there is inherent appeal in the a-paradigmatic approach, where the focus shifts away from paradigmatic assumptions towards the suitability of methods, thereby thwarting a oneto-one relationship between paradigm and methods (Archibald, 2016; Greene, 2007). However, a definitive focus on methods negates the influence that theory-driven disciplinary socialization has on research approaches, thereby overlooking the influence of epistemic culture. It also risks overlooking traditional communicative and organizational structures within disciplines that emphasize hierarchical leadership and individual accomplishment (Knorr-Cetina, 1999). As such, investigators engaged in transdisciplinary research using an a-paradigmatic stance might encounter barriers to collaborative work—including efforts at integration—as implicit assumptions are likely to surface during the collaborative process. Conversely, the substantive theory stance requires investigation of the theories held by investigators and could aid in a communal identification of the nature of the problem or problem components necessary to facilitate true transdisciplinary inquiry. Similar to the nested concept purported by Morgan (2007), a substantive theory stance could facilitate an alternative paradigm stance where a new perspective is generated in response to context and theory specific demands. This generative production aligns well with the aims of transdisciplinary research, which recognizes the value of new conceptual and methodological frameworks (Klein, 2018). Such context- and theory-driven demands are reminiscent of the “dynamic interplay between theory and practice or between thinking/knowing and acting/doing … [which] is actually a hallmark of Deweyian pragmatism” (Greene, 2008, p. 8). Indeed, pragmatism has long since been considered a “leading contender for the philosophical champion of mixed methods research” (Greene, 2008, p. 8) and holds similar appeal as a champion for transdisciplinary mixed methods research (Bergmann et  al., 2012). As such, while other paradigmatic perspectives could arguably underpin a transdisciplinary mixed methods research project or programme, attention will be given here to pragmatism as the most likely candidate for paradigmatic guidance in transdisciplinary mixed methods research.

Pragmatism and Reflexive Questioning for Transdisciplinary Mixed Methods Research Pragmatism has been conceptualized as a problemsolution oriented approach that emphasizes communication and shared meaning making in

515

the practical pursuit of answering a socially relevant research problem (Johnson & Onwuegbuzie, 2004; Shannon-Baker, 2016). The classical pragmatists (e.g., John Dewey, William James, Charles Sanders Peirce) forwarded that knowledge about the world is inextricably tied to the agency within it, and hence the utility of claims in relation to social impact becomes a central consideration (Legg & Hookway, 2021). Given that transdisciplinary research is inherently rooted in the understanding and response to real-world social problems, there is notable congruence between transdisciplinary mixed methods and a pragmatic stance to research, wherein the merit of an idea is evaluated on the basis of its empirical and social consequence. In the context of mixed methods research, Johnson & Onwuegbuzie (2004) discuss that it is the pragmatic maxim that is used “to determine the meaning of words, concepts, statements, ideas and beliefs” and determine “what effects the object of our conception may have” (p.16). This notion of the object of conception takes particular precedent in the context of transdisciplinary mixed methods in relation to design (which will be discussed later on), but also provides a bridge to the agency and contextually bound nature of perceptual objects within pragmatism. Specifically, the epistemic object (i.e., those issues that can be recognized and subjected to study) within a transdisciplinary study must be created through decoupling of the object from their contextual (and disciplinary) foundations (Bergmann et al., 2012). Such decoupling requires an understanding of disciplinary constructed understandings and theories, reminiscent of the substantive theory stance previously discussed, as well as the combination of understandings in a new problem conceptualization, suggestive of an alternative paradigm stance. Helpful to this decoupling is the recognition of a boundary object—an interface wherein such collaborative discussion and ideation can occur. Working with a boundary object may involve materializing an objective and grounding it in a particular service, product or artefact (Bergmann et  al., 2012). The resulting concrete objective helps disentangle the epistemic object, creating a collective orientation and bridge across disciplines and between scientific and non-scientific actors (Bergmann et  al., 2012), and, like pragmatism more generally, shifts inquiry from an exclusively theoretical emphasis towards one grounded in real-world effects. Yet, the complexities associated with the decoupling of epistemic objects from their disciplinary contexts and reconceptualizing the nature of the problem in the light of such understandings underscore that deconstructing one’s own perspective is necessary prior to contributing to such

516

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

a collective endeavour. It follows that reflexivity, and specifically reflexive questioning, is critical to this foundational process, particularly since transdisciplinary research itself is understood as a self-reflexive and critical approach merging social and scientific knowledge. Reflexive questioning involves critical inquiry into one’s “values, assumptions, and normative orientations” (p. 46) and is regarded by Popa and colleagues (2015) as a pragmatic approach to reflexivity. This approach positions reflexivity within the pragmatic practice of transdisciplinary research through “participation in concrete problem-solving and social experimentation and learning processes” (p. 46) and offers a practice-based approach to the identification of assumptions governing problem identification and conceptualization. Reflexive questioning can facilitate the unveiling of paradigmatic perspectives as constructed—rather than reified and essential (Freshwater & Cahill, 2013)—underscoring their influence on methodology while also illuminating those often overlooked valueladen orientations influencing which problems are regarded as relevant, how such problems are best approached and how methods are valued (or not) within a research endeavour. Such an approach could enable investigators to individually and collectively identify theories and assumptions held prior to commencing the shared conceptual work required of transdisciplinary mixed methods research, after engaging in initial dialogue around the determined boundary object. As such, pragmatism and its functional bedfellow, reflexive questioning, may indeed be precursors, if not merely facilitators, to the predominant objective of transdisciplinary research: the integration of various scientific and social bodies of knowledge.

COLLABORATIVE APPROACHES AND INTEGRATION: DISTINGUISHING TRANSDISCIPLINARY MIXED METHODS RESEARCH APPROACHES Multidisciplinary, Interdisciplinary, Transdisciplinary Research Multidisciplinary, interdisciplinary and transdisciplinary: these terms reflect a sentiment that working together can strengthen research practices and help investigators (e.g., scientists, researchers) move beyond the confines of their own discipline ideally to develop better solutions to pressing problems (Mitchell, 2005). Although these terms are often used interchangeably, they carry distinct

meanings, and can be thought of on a continuum of disciplinary integration. At one end of the continuum is multidisciplinary research, which involves investigators from more than one discipline who work together but maintain the “integrity” of their own disciplinary knowledge (Rosenfield, 1992). Here, researchers work in parallel or sequentially to address a common problem (Mitchell, 2005). Interdisciplinary research resides in the middle of this continuum and involves a synthesis, bridging or linking of disciplinary knowledge. Here, researchers work jointly, but maintain their disciplinary knowledge bases (Mitchell, 2005). At the fully integrative end of the continuum resides transdisciplinary research, which involves each discipline moving past the confines of their disciplinary knowledge to create something not possible without this degree of collaborative partnership (Bergmann et  al., 2012; Klein, 2018). Here, researchers approach a problem from a shared conceptual framework that draws upon and integrates diverse sources of disciplinary knowledge (Fam et al., 2018). If each collaborative approach was offered an equation or numerical representation, we might consider the following. Multidisciplinary research is essentially non-summative. Two or more disciplines can be identified, but the result is theoretically non-integrative (e.g., 1, 1). Interdisciplinary research provides to this non-summative count the required action (i.e., the addition sign, signifying bridging or synthesis of disciplinary knowledge), enabling a summative resolution between the two tallied numbers. For instance, interdisciplinary collaboration involving two disciplines provides the + between the 1, 1, enabling a solution of 2 to be achieved (1 + 1 = 2). However, transdisciplinary research moves beyond the confines of each respective discipline to create a solution that exceeds the summative potential of each respective number, resulting in a representation of 1 + 1 = 3, reminiscent of the integration challenge in mixed methods research (Fetters & Freshwater, 2015).

Mixed Methods Research and the “Integration Challenge” Mixed methods research is understood as the combination of qualitative and quantitative research approaches, by an individual or team of investigators, with the intent of increasing understanding of the research problem (Johnson et al., 2007). Creswell and Plano-Clark (2011) emphasize that mixed methods research involves three dimensions, including methods, a philosophy and a research design orientation. Greene (2008)

OPPORTUNITIES AND CHALLENGES FOR A TRANSDISCIPLINARY FUTURE

discusses that mixed methods research involves a mixed methods way of thinking—a dialogical movement appreciative of both contextual specificities and distant generalities. Despite some diversity regarding emphasis in these, and other, definitions of mixed methods research, a predominant commonality and emphasis is integration. Specifically, mixed methods research involves the meaningful bringing together of qualitative (narrative) and quantitative (statistical/numerical) data in a single study or programme of study. Integration in mixed methods research can carry different meanings related to what is being brought together, when and how. Integration itself has been described as merging, blending, braiding, combining and synthesizing (Bazeley & Kemp, 2012; Watson, 2020). The timing of integration and what exactly is being mixed are also imperative considerations that both flow from and inform research design. Paradigms, philosophical stances, methodology, research samples and research data are common examples of what is mixed, combined and so forth in a given study (Bryman, 2006; Collins et al., 2007; Creswell & Plano Clark, 2011; Teddlie & Yu, 2007). Despite the persistent interest in that what and how of integration that has in many ways characterized mixed methods research, the “who” of integration—that is, interest in the disciplinary mergers and mixtures, as well as the mergers and mixtures of individuals working within mixed methods projects and teams—has received less attention (Archibald, 2016). A few exceptions can be observed in the mixed methods literature, yet their emphasis is almost exclusively on interdisciplinary rather than transdisciplinary research. For instance, Lunde and colleagues (2013) explored what they term “unproductive interplay” between qualitative and quantitative researchers in a mixed methods team study. Although the work proceeded within an interdisciplinary, rather than transdisciplinary framework, the authors’ findings that power differentials, material imbalances and academic hierarchies were barriers to integration of research findings are pertinent to a transdisciplinary mixed methods research agenda. Also critical is that a clinically driven research problem conceptualization that did not involve the entire team may have impacted opportunities for integration through perceptions disempowerment, which further impacted team functioning. O’Cathain and colleagues (2008) identified three models of team work through semi-structured interviews with researchers working on mixed methods teams, including multidisciplinary, interdisciplinary and dysfunctional. They identified leadership and methodological respect as critical to effective team functioning; transdisciplinary research did not surface as a predominant model in their analysis.

517

Hemmings and colleagues (2013) explored team dynamics, again within an interdisciplinary rather than transdisciplinary framework, and emphasized team socialization and re-education as prerequisites for group social identity and effective work at paradigmatic, methods and technique levels. Similarly, Curry et al. (2012) emphasized the challenges and mitigating strategies related to interdisciplinary mixed methods research, highlighting dealing with difference, trusting the other, creating a meaningful group, handling essential conflicts and effective leadership in their discussion. More recently, Hesse-Biber (2016) highlighted the need for attention to team dynamics and higher levels of reflexivity to promote effective interdisciplinary team functioning. The emphasis on the “what” and the “how” of integration in mixed methods research, rather than the “who” of integration, suggests an historical emphasis towards practical, problem-driven uses of qualitative and quantitative methods. It also renders the human influence invisible, negating how socialization, organization and interaction within teams can influence design decisions and execution. Indeed, evidence of the concurrent or sequential use of qualitative and quantitative research methods has been documented in investigations dating back to the nineteenth and twentieth centuries and even earlier (e.g., Galileo in the 1600s; Maxwell, 2016), but mixed methods did not emerge as a self-aware discipline until the formative period of mixed methods in the 1950s (Creswell & Plano Clark, 2011). This history is important because long before the self-identification of mixed methods as a distinct methodology, researchers from health, social and evaluative sciences were driven to combine qualitative and quantitative methods out of necessity—that is, the complexities of the problems under investigation required contributions of both qualitative and quantitative methodologies (Greene, 2008). Similarly, awareness of “wicked problems”— those persistent social problems that are characterized by evasiveness to investigation and change due to their complexity (Rittel & Webber, 1973)—has increasingly prompted researchers to look towards collaborative approaches. These approaches help move beyond the parallel contributions of different disciplines (i.e., multidisciplinary) towards the synergistic contributions made possible through transdisciplinary research. As such, while mixed methods research can be regarded as a response to the requirements of understanding social complexity, transdisciplinary research and, specifically, transdisciplinary mixed methods research, can be seen as the next step in this evolution and, critically, is driven by the need for practical solutions to complex social problems.

518

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

PRACTICAL GUIDANCE: THE INNER WORKINGS OF TRANSDISCIPLINARY MIXED METHODS RESEARCH Dimensions of Integration in Design Just as integration in mixed methods research can refer to myriad time points and sources (e.g., data, methods, methodology), integration in transdisciplinary research crosses multiple dimensions. Integration is most often assumed to have an exclusively knowledge-based or epistemic premise, with authors at times referring to the disciplinary and knowledge production (mode) aspects of integration (Hesse-Biber, 2016). Yet, integration can occur across other interrelated dimensions. According to Bergmann et al., 2012, these dimensions include: (1) the communicative dimension, wherein linguistic and communication preferences and practices are differentiated and linked, and a common discursive practice is established based on this emergent shared understanding; (2) the social and organizational dimension, wherein the different interests of researchers and associated sub projects are considered, differentiated between and correlated, and where deliberate leadership is critical to achieving such mutual understanding and alignment; and (3) the cognitive-epistemic dimension, wherein disciplinary knowledge bases are differentiated and linked and real-world impacts are considered. Types of integration can cut across these three dominant integrative dimensions. For instance, symmetric integration involves the complementary contribution of disciplinary knowledge to the understanding and results produced about a specific research problem. Such differentiation enables gaps to be identified in understanding and approach, and requires notable decomposition of the research problem from the onset of research for symmetric integration to occur (Bergmann et  al., 2012). In contrast, integration across the social and natural sciences presents challenges due to the heterogeneous knowledge bases that each occupy; rather than achieving a complementary integration, they, through mutual dependency, “mutually reveal each other’s constraints” (p. 46) (Bergmann et  al., 2012). For all transdisciplinary mixed methods designs, motivation is derived from societal problems of interest, thereby requiring integration of scientific and societal bodies of knowledge (Bergmann et al., 2012). Regardless of the type and extent of integration required of researchers working at the intersection of transdisciplinary and mixed methods research, establishing a base understanding of theoretical perspectives guiding the research is a paramount

consideration, as well as a notable undertaking. This recognition is required at multiple levels, including the levels of the individual researcher, and identifying the theoretical orientation of research concepts according to each discipline. Here, the concept of “representational groups”, as forwarded by Curry et  al. (2012), is relevant: individuals in a group represent both organizational and professional groups (e.g., disciplines, associations), but also their identity groups (e.g., gender). Reflexive questioning to identify positioning can be followed by subsequent dialogue within the team regarding each theoretical perspective, as well as the influences that these have on the research topic (concept/s) under study. To these ends, Alavi and colleagues (2018) offer the Before Design Theoretical Placement (BDTP) as a means of assisting researchers in identifying and locating their research within broader theoretical grounding. The authors propose a relational metatheoretical stance drawing from qualitative and quantitative terminology, to communicate that different methodologies reflect respective perspectives of a unified whole (Alavi et al., 2018) They offer a stepwise approach to understanding the theoretical positioning of the researcher and the theoretical components of the research concepts, recognizing that such bases impact research methods in oft under acknowledged ways. Consider the impact that such theoretical work, inclusive of the decomposition of the research problem, could have made to the interdisciplinary mixed methods research of Lunde et al. (2013), who documented a lack of leadership and a quantitatively driven research problem conceptualization as barriers to integration within their study. Once an understanding of the theoretical basis of the research concept has been established, the nature of the disciplinary knowledge that has led to its operationalization within a discipline is possible. Within transdisciplinary teams however, establishing a theoretical understanding of the research concept is markedly more complex than occurs within a single disciplinary study. In part, this is because “all theoretical knowledge is disciplinary knowledge” (Bergmann et  al., 2012, p. 56), requiring that such knowledge be uncovered, and its methods critically examined, as a preliminary requirement for the conceptual work that is required of teams seeking conceptual clarity (and conceptual integration) of the research concept. The research methods that give way to this understanding may not be understood across disciplines; the scaffolding of subsequent knowledge within a discipline may have taken a different orientation than the scaffolding of another discipline (e.g., on the same topic of study; consider for instance, the concept of illness as understood

OPPORTUNITIES AND CHALLENGES FOR A TRANSDISCIPLINARY FUTURE

by a phenomenological qualitative research in the health sciences as opposed to a medical sociologist; a medical practitioner trained in western medicine; or, an Ayurvedic practitioner). The conceptual work required by a transdisciplinary team is a necessary precursor to any transdisciplinary mixed methods project, and carries substantial impact on the research methods, research design, and ultimate legitimacy and impact of the research project(s) undertaken.

Integrative Conceptual Work within Transdisciplinary Mixed Methods Research Transdisciplinary research is driven by socially relevant problems of the “real world”; problems so complex that they require the multidimensionality that only various disciplines can provide (Bergmann et  al., 2012). Before mixed method designs can be established however, critical conceptual work within the transdisciplinary team must be undertaken. Bergmann and colleagues (2012) present this conceptual work as comprising two stages. The first stage requires that all research concepts are identified and defined. Such identification of core concepts takes place through team discussion. As such, this first stage requires an interpersonal dynamic alongside other factors, to render it successful. Leadership and associated strategies to foster trust and respect, often through deliberate communication and facilitation, have been shown to be key factors in establishing group identity and meaningful team cooperation in transdisciplinary (Archibald et  al., 2023) and interdisciplinary mixed methods research (O’Cathain et al., 2008; Curry et al., 2012; Lunde et al., 2013). Organizational structure that emphasizes collective rather than individual identity may reduce competition between investigators and promote a collaborative orientation to the research (Knorr-Cetina, 1999). Importantly, this stage may require new terms or concepts to be developed to reflect the crosscutting concepts and real-world problems under question. After the concepts have been identified and the new concepts developed as required, the second stage of conceptual integration tasks researchers with linking each concept to the disciplinary specific terms and concepts in a manner that preserves the integrity of the larger social problem in question (Bergmann et  al., 2012). Such conceptual work is critical to overcoming the disciplinary silos and tendency to revert to working within such silos, which is a significant threat to the optimization of transdisciplinary mixed method research.

519

The necessity of this conceptual work cannot be overstated. Not only does it prevent researchers from working within their disciplinary specific approaches at the expense of the shared research goal, it enables the identification of a shared research goal to begin with (and corresponding understanding of the nature of the research problem). Further, it permits each concept to be clarified according to the understandings of each involved discipline. Agreement of the terms is more than an epistemic exercise and can have considerable impact in how each section of the team accepts the knowledge constructed, how such knowledge is then built upon, and whether the knowledge and research findings are ultimately used (or not) to influence practice, services or policy. Similarly, since concepts may be unfamiliar to individuals on the team, and some concepts may have only been examined in a “narrow, disciplinary specific manner” (p. 61), these cognitive exercises necessarily provide access points for the team as they move towards methods of improving understanding and impact of the broader research problem. Not attending to the value-laden nature of how problems arise, are socially recognized and then identified (and tackled) by researchers has implications for research conceptualization, and ultimately determines what gets selected as a relevant domain of study by the mixed methods research community (Popa et  al., 2015). Collective problem conceptualization is therefore a foundational design element to any transdisciplinary mixed methods design; unilateral conceptualization has been documented as a barrier to empowerment within interdisciplinary research teams impacting team functioning (Lunde et al., 2013). Conversely, a collective sense of ownership through shared vision, problem and project formulation can facilitate engagement, investment and multidimensional input from team members (Archibald et al., 2023) thereby contributing to team functioning (process), and improving the likelihood of scientific and social knowledge contributions.

Practical Strategies for Teams in Transdisciplinary Mixed Methods Research In addition to the philosophical considerations, which may indeed have lasting procedural impacts, there are a number of practical considerations related to the human aspect of transdisciplinary research worthy of consolidation. Some of these considerations are drawn from interdisciplinary and team-based mixed methods literature. Others are extrapolated from transdisciplinary

520

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

methodological writing and applications in other arenas. Others still result from my experience as a mixed methods researcher with experience working in transdisciplinary teams, including a meta study of transdisciplinary research for impact (Archibald et al., 2023; Archibald et al., 2018). Transdisciplinary research is inherently methoddriven (Popa et  al., 2015). While it is tempting, then, to assume that philosophical considerations do not apply, failing to recognize the philosophical underpinnings of investigators risks having these implicit assumptions surface in research decisions (e.g., Which problem is important? Which methods are the most appropriate to address the question? Which findings should be prioritized?) and interpersonal dynamics. Encouraging adaptability and openness to the social problem of interest, as well as the approaches taken to address it, is therefore necessary. Reflexive questioning (Popa & Guillermin, 2017) may be fruitful in helping identify assumptions and creating the groundwork for a consolidated vision of the research problem, methods and other design-related decisions. Awareness that the research design may be emergent given new understandings of the nature of the problem may reduce individual resistance to adaptable design. Investigators often have diverse epistemological positioning, and keeping a “broad view” is challenging (Deschepper et  al., 2017); clashes can occur between epistemic communities (Knorr-Cetina, 1999). As such, leadership, methodological respect, creating a meaningful group or shared identity and perceived investment (engagement) are critical to the eventual success of the transdisciplinary mixed methods team (Archibald et  al., 2018; Archibald, 2016; Curry et al., 2012; Hemmings et al., 2013; Lunde et  al., 2013; O’Cathain, 2008). An “effective” team is one that deals well with conflict, often by appealing to the shared objective (Archibald et  al., 2023), and through enactment of leadership. Trust is understood as a mechanism enabling team progress; such trust can commence as transactional, incrementally built through reciprocal delivery and exchange of quality research outputs or other outcomes of interest to a team (e.g., timely communication), or can be linked to scholarly standing (Archibald et  al., 2023). Leadership, however, is best understood as a role rather than a position associated with hierarchy, or as an individual characteristic (Curry et  al., 2012). Hierarchical leadership may be a barrier to the more collective identity noted in more collaborative organizational structures (KnorrCetina, 1999). As Curry et al. (2012) emphasize, these factors intersect in meaningful ways rather than existing as exclusive categories.

Indeed, creating a well-functioning transdisciplinary group involves a number of interrelated factors. Within a transdisciplinary mixed methods project, this involves potentially prolonged conceptual work to establish a shared vision of the problem of interest, and the associated materials that reflect the transdisciplinary understanding. The need for a common language and set of terms in transdisciplinary mixed methods exceeds a similar requirement noted in the team mixed methods literature, where authors have called for tangible items such as a glossary of terms (Curry et  al., 2012). Given the context dependency of any transdisciplinary project, associated terminologies (which reflect specific understandings of the problem contingent on the composition of the team designed to address them) and methods “must be decoupled [decontextualized] from their original contexts – and described in general terms” (Bergmann et  al., 2012, p. 15). Methods, such as using heuristics like metaphors to examine a problem of interest, considering boundary objects (e.g., objects with loose meanings in different social contexts that can aid interdisciplinary access to a problem), agreeing on a shared conceptual framework, and engaging in a double-sided critique of naturalistic and culturalistic approaches to research in order to distinguish scholarly origins of understandings and soften epistemic differences, are critical to establishing the conceptual foundation for transdisciplinary mixed methods (Bergmann et  al., 2012). However, this conceptual work is recursive rather than linear, requiring revisiting as the project progresses. Strong team identity, including individual investment and engagement in the process, as well as openness to design modifications, are therefore necessary.

NEW POSSIBILITIES FOR FUTURE MIXED METHODS RESEARCH DESIGNS Looking forward to what the future of transdisciplinary mixed methods research might look like requires a concurrent focus on the multiple domains of Greene’s (2008) framework, with a renewed emphasis on Lunde and colleagues’ (2013) addition of human factors. Post facto recognition of human influence in transdisciplinary mixed methods will be insufficient to optimize the effective function and influence of research design. As transdisciplinary mixed methods become more common, attention to and deliberate planning of team structure and organization will be necessary to capitalize on the full potentials of transdisciplinary mixed methods research.

OPPORTUNITIES AND CHALLENGES FOR A TRANSDISCIPLINARY FUTURE

Achieving this objective will be aided through an understanding of realist principles, to provide nuanced understanding of what is working and why within a team, as well as the application of this understanding in tailored facilitation to maximize team function. Similarly, as transdisciplinary mixed methods research designs proliferate, previously untapped disciplinary mergers will surface, providing new challenges and opportunities impacting understanding, analytical procedures and research design. Communication and technological considerations will be instrumental in supporting these efforts, offering benefit to both the operational aspects of transdisciplinary mixed methods, as well as supporting future collaborative initiatives. Offered here are foreseeable opportunities for such a transdisciplinary mixed methods future, as well as identification of plausible challenges and associated mitigation strategies. While the possibilities are many, this discussion focuses on the potentials of realist principles and middle-range theory, innovative disciplinary mergers, communication and technological considerations, and recursive design.

Realist Principles and Tailored Facilitation In many ways, transdisciplinary research has been a response to the nature of wicked problems. As such, the response to this complexity mandates a response equally grounded in complexity science. This response should be reflected in both the philosophical (epistemic, ontological and axiological) approach underpinning the work, but also manifested in the approach to the research design. Transdisciplinary team research that draws upon middle-range theory, such as theory derived through realist evaluation, can provide context specific guidance pertinent to transdisciplinary team research (Archibald et  al., 2018) and can provide a nuanced evidence-based method of bolstering transdisciplinary team research and effectiveness. While some have argued that we fail to do interdisciplinary mixed methods “well”, and that shortcomings may be linked to a lack of understanding about what promotes “vibrant team dynamics” (Hesse-Biber, 2016, p. 650), there is a strong imperative for the development, testing and use of theoretically driven strategies to understand and promote effective transdisciplinary mixed methods research. The next five to ten years might encounter the utilization of such middle-range theory, and fundamentally it’s testing in different contexts. Use of realist evaluation logic (what works for whom, in what contexts, to what extent,

521

and how; Pawson & Tilley, 1997) would enable identification of mechanisms and contexts that positively and/or negatively impact transdisciplinary mixed methods research, in relation to methodological outcomes of interest (e.g., extent of integration), as well as social outcomes. It follows that, after a more intentional transdisciplinary mixed methods strategy, the future might benefit from a deliberate mapping and titration of strategies known to be effective for bolstering teams in particular contexts. Data detailing team composition in relation to particular problem domains, or even across various disciplinary compositions, may be generated, likely again in the form of middle-range theory, and may help inform optimal team composition or, at a minimum, inform key domains informing team construction. We are also likely to see further influence of realist ontology and/or realist principles (including realist evaluation) in transdisciplinary mixed methods research design, to match the growing recognition that complex problems require complex solutions, and the mounting interest in the explanatory potential of realist evaluation. Concurrently, publishing transdisciplinary mixed methods will require a team disclosure, a comprehensive description of how the conceptual work of the team was completed, and to “what ends”. This disclosure will go beyond a cursory description of the team to enable understanding of the scientific approach and grounding, thereby contributing to the science of transdisciplinary work, as well as the practical benefit of conducting such research.

Innovative Mergers Underexplored mergers made possible through transdisciplinary mixed methods abound. Broader mergers of disciplines, such as those emphasizing other forms of knowledge and ways of being, such as artistic practices, may provide entirely new insights and ways of approaching problems. Growing interest in arts-based mixed methods research suggests this may become more popular, particularly as team proficiency improves and the methodological practices become more established (Archibald, 2018; Archibald & Gerber, 2018). Such practices may have particular appeal for the decoupling of disciplinary knowledge as the arts can help move thinking beyond the limitations of the discursive, enabling new ways of approaching and considering a problem through new means of conceptualization (aiding in problem conceptualization) (Archibald, 2021). Further, the integration of subjective and objective forms of understanding, such as community-led

522

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

emergent research designs and the systematic involvement of researchers as autoethnographic subjects (i.e., systematic study (graphy) of the self (auto) in relation to culture (ethno), but the extent to which individuals focus on the self, others, culture and power may differ; Ellis et al., 2011). Mergers between the basic, applied and social sciences, which currently are apt for development, are another likely area of growth in transdisciplinary mixed methods research. Such mergers could provide new opportunities for integration across communicative, organizational and social, as well as cognitive-epistemic dimensions (Bergmann et al., 2012).

Communication and Technological Considerations Regarding research design, transdisciplinary mixed methods research should leverage visual approaches to communicating about research design (e.g., Shannon-Baker & Edwards, 2018) as well as notation systems to help communicate the historical progression of conceptualizations, methodological, methods and technique-based decisions within a team as they evolve. Moving beyond the use of visual methods for communicating research designs, this strategy may foster understanding between disciplines; help conceptually map terms, their lineage and approaches to study in complement to a glossary of terms (cf. Curry et  al., 2012); and enable transparency in reporting as well as external appraisal of the conceptual trails followed and developed in response to decontextualizing and reconceptualizing the epistemic object within the transdisciplinary framework. Technological support for such processes and representations vis-à-vis mixed methods software will be imperative, as will the use of such software in the support of team-based analysis. Software features that allow the linking of analytics to theoretical constructs, linking these to disciplinary understandings and group-derived shared conceptual frameworks, may help advance understanding of how such conceptual work occurs in a mixed methods team. Existing software features, such as logbooks within MAXQDA, for example, can assist teams in accurately documenting research progress, insights and proceedings; each user’s logbook entry can be used to facilitate and accompany team-based analytic and dissemination efforts (Kuckartz & Rädiker, 2019). Technology can also assist in understanding the extent of agreement between investigators participating in data analysis. While the importance of such agreement

analysis is contingent on the research design, intercoder analysis can help sharpen category definitions and coding instructions, thus improving communication and systematicity when multiple investigators are involved (Kuckartz & Rädiker, 2019). This is likely to be of particular importance when diverse conceptualizations of research phenomena are at play, as is common within transdisciplinary mixed methods team contexts. In addition to supporting the operational components of transdisciplinary mixed methods research, there is notable potential for technology to help teams organize and cohere in new ways. Continued advancement and use of video conferencing software, such as Zoom, has reduced geographic barriers inhibiting multisite team research (Archibald et  al., 2019). Although such text and video-based software has recently gained significant acceptance as a research tool, the future may witness opportunities for more interactive platforms to support transdisciplinary mixed methods research. For instance, virtual reality-based collaboration may enable multiple speakers to engage simultaneously over a shared topic at one time, emphasizing the linearity of the solo-speaker model of traditional video-conferencing platforms and highlighting its shortcomings. These opportunities have begun to garner attention in areas such as building design (Ojala et al., 2020), but reflect untapped potential for transdisciplinary mixed methods research. Further benefits of virtual reality for fostering transdisciplinary mixed methods research may relate to the ability to orientate individuals to a particular perspective or location (Albaek Thomsen et  al., 2019), to facilitate the shared conceptual work and vision required for effective transdisciplinary team research.

Recursive Design Progressively over the next five years, but increasingly over the next 15 years, the likely proliferation of transdisciplinary mixed methods research will catalyze a new interest in recursive designs. The concept of recursion is often applied to mathematics and is understood as the repeated application of a rule or approach (Merriam-Webster Dictionary, 2021). In the research context, recursivity holds familiarity; it surfaces in reference to scholarly writing, where ideas are continually constructed, revised and revised (Hudd & Bronson, 2007). It is characteristic of qualitative research—namely, the iterative process of data collection and analysis—wherein analytic findings may inform the need for subsequent data collection until a particular condition (e.g.,

OPPORTUNITIES AND CHALLENGES FOR A TRANSDISCIPLINARY FUTURE

theoretical saturation) is satisfied (Given, 2008). Recursivity is pertinent to mixed methods and is evidenced by growing acknowledgement of its dynamic and undetermined nature (Archibald, 2016; Poth, 2018). Here, recursivity has been discussed by Christ (2007) in reference to how interim analyses can inform mixed method design modifications, and who emphasizes the importance of methodological flexibility. Sanscartier (2020) provided a contemporary compliment to this discussion and argues how a craft attitude, inclusive of “comfort with uncertainty, a nonlinear/recursive approach to research, and understanding research as storytelling” (p. 47), can help navigate two dominant forms of mess in mixed methods research. This includes mess of the empirical form, wherein the results from qualitative and quantitative investigations may or may not converge, and mess of the design form, where designs are modified continually to attend to context and data-based considerations. Within the context of transdisciplinary mixed methods, however, and in reference to the critical, foundational conceptual work required of the team early in problem conceptualization, recursivity is necessary to navigate the differing conceptualizations of the research object and to revisit its framing as the project progresses. Indeed, the emergent and iterative nature of recursivity is particularly pertinent to transdisciplinary mixed methods research, given the complexities of the (real-world) research problem, teams and combinations of disciplines and methods involved in its study. While it is not uncommon for qualitative researchers to revisit their research problem framing following the collection of data (Given, 2008), this practice is all but necessary within pragmatic, self-reflexive transdisciplinary mixed methods frameworks, which are unpredictable and emergent by nature. A commitment to recursive design is reminiscent of Bryman’s (2006) early critique that traditional mixed methods typologies suggest a commitment to a particular type of data and a level of predictability. This predictability is contrary to the complexity and emergent nature of transdisciplinary mixed methods research, which is more aptly guided by understandings of effect. For instance, and as aligned with the pragmatic emphasis discussed earlier on, these recursive research approaches may involve the periodic implementation of emerging findings; cyclical implementation may be built into the transdisciplinary research process rather than indicating an end-point, or product, of the mixed methods research endeavour (Bergmann et  al., 2012). Resultantly, robust documentation of various recursive design possibilities are likely to follow in the next 10 to 15 years, and are likely to

523

require more advanced notation systems or visual depictions of design and implementation developments, likely aided by technology.

CONCLUSION In this chapter, I have argued for transdisciplinary mixed methods research as a necessary approach for attending to wicked social problems, and have highlighted its opportunities and key considerations. It is well recognized that mixed methods research provides a methodological bridge for integration of qualitative and quantitative research approaches, thereby offering a form of transcendence of methodology (and a generative offering of a third, mixed methods methodology). Similarly, transdisciplinary research is regarded as an approach to overcome disciplinary silos, or boundaries. The combination of transdisciplinary research and mixed methods research has the synergistic potential to transcend both disciplinary and paradigmatic boundaries and, in doing so, produce a unique research position generated as a byproduct of team composition and in response to a particular real-world problem. However, the potential of transdisciplinary mixed methods research is largely contingent on the human factors influencing research design, and requires attentive conceptual work and leadership, while appealing to a shared vision and investment in social impact. Ensuring such a basis, including a grounding in one’s own philosophical assumptions and willingness to engage in the theoretical groundwork necessary to decouple epistemic objects from their disciplinary context, is necessary. A pragmatic, substantive theory stance inclusive of reflexive questioning may be a useful bedfellow for a transdisciplinary mixed methods research agenda.

WHAT TO READ NEXT Archibald, M. (2016). Investigator triangulation: A collaborative strategy with potential for mixed methods research. Journal of Mixed Methods Research, 10(3), 228–250. https://doi.org/10.1177/ 1558689815570092

This article provides insights into the generative potentials of collaboration in mixed methods specific to the deliberate bringing together of diverse investigators. As such, it is highly relevant to the transdisciplinary mixed methods context;

524

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

considerations regarding quality and tensions emerging from collaboration may be particularly useful for readers. Bergmann, M., Jahn, T., Knobloch, T., Krohn, W., Pohl, C., & Schramm, E. (2012). Methods for Transdisciplinary Research. A Primer for Practice. Campus Verlag.

This detailed text provides solid grounding in transdisciplinary research methods, including types of integration, specific methods and other key pathways for conducting transdisciplinary research. In addition to this foundational text, Bergmann has authored and co-authored a number of pertinent scholarly articles worthy of perusal. Fam, D., Neuhauser, L., & Gibbs, P. (2018). Transdisciplinary theory, practice and education: The art of collaborative research and collective learning. In Transdisciplinary Theory, Practice and Education: The Art of Collaborative Research and Collective Learning.

This book draws from an abundance of international contributors, emphasizes the production of science with society, and provides a strong theoretical background to the field. A particularly useful contribution of this text, however, is the emphasis that it places on the teaching of transdisciplinary approaches, and the range of case examples reflecting a broad combination of disciplines, including art/science interfaces and Indigenous perspectives.

REFERENCES Alavi, M., Archibald, M., McMaster, R., Lopez, V., & Cleary, M. (2018). Aligning theory and methodology in mixed methods research: Before Design Theoretical Placement. International Journal of Social Research Methodology, 21(5). https://doi. org/10.1080/13645579.2018.1435016 Albaek Thomsen, L., Nordahl, R., Nilsson, N. C., & Lohmann, B. (2019). Asymmetric collaboration in virtual reality: A taxonomy of asymmetric interfaces for collaborative immersive learning. Læring Og Medier, 12(20). https://doi.org/10.7146/lom. v12i20.109391 Archibald, M. (2016). Investigator triangulation: A collaborative strategy with potential for mixed methods research. Journal of Mixed Methods Research, 10(3), 228–250. https://doi.org/10.1177/ 1558689815570092 Archibald, M. (2018). Integrating the arts and mixed methods research: A review and a way forward. International Journal of Multiple Research

Approaches, 10(1), 342–355. https://doi.org/ 10.29034/ijmra.v10n1a23 Archibald, M. (2021). Arts and mixed methods research for social inclusion. In Handbook of Social Inclusion Research. In Liamputtong P. (eds) Handbook of Social Inclusion. Springer. https://doi. org/10.1007/978-3-030-48277-0_20-1 Archibald, M., & Gerber, N. (2018). Arts and mixed methods research: An innovative methodological merger. American Behavioral Scientist, 62(7), 956– 977. https://doi.org/10.1177/0002764218772672 Archibald, M. M., Lawless, M., Harvey, G., & Kitson, A. L. (2018). Transdisciplinary research for impact: Protocol for a realist evaluation of the relationship between transdisciplinary research collaboration and knowledge translation. BMJ Open, 8(4). https://doi.org/10.1136/bmjopen-2018-021775 Archibald, M. M., Ambagtsheer, R. C., Casey, M. G., & Lawless, M. (2019). Using Zoom Videoconferencing for Qualitative Data Collection: Perceptions and Experiences of Researchers and Participants. International Journal of Qualitative Methods, 18. https://doi.org/10.1177/1609406919874596 Archibald, M. M., Lawless, M., Pinero de Plaza, M., & Kitson, A. (2023). How transdisciplinary research teams learn to do knowledge translation (KT), and how KT in turn, impacts transdisciplinary research: A realist evaluation and longitudinal case study. Health Research Policy and Systems. 10.1186/ s12961-023-00967-x Bazeley, P. (2012). Integrative analysis strategies for mixed data sources. American Behavioral Scientist, 56(6), 814–828. https://doi.org/10.1177/00027642 11426330 Bazeley, P., & Kemp, L. (2012). Mosaics, triangles, and DNA: Metaphors for integrated analysis in mixed methods research. Journal of Mixed Methods Research, 6(1), 55–72. https://doi. org/10.1177/1558689811419514 Bergmann, M., Jahn, T., Knobloch, T., Krohn, W., Pohl, C., & Schramm, E. (2012). Methods for transdisciplinary research. A primer for practice. Campus Verlag. Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6(1), 97–113. https://doi.org/10.1177/146879 4106058877 Christ. T. W. (2007). A recursive approach to mixed methods research in a longitudinal study of postsecondary education disability support services. Journal of Mixed Methods Research, 1(3), 226–241. https:// doi.org/10.1177/1558689807301101 Collins, K. M. T., Onwuegbuzie, A. J., & Jiao, Q. G. (2007). A mixed methods investigation of mixed methods sampling designs in social and health science research. Journal of Mixed Methods Research, 1(3), 267–294. https://doi.org/10.1177/1558689 807299526

OPPORTUNITIES AND CHALLENGES FOR A TRANSDISCIPLINARY FUTURE

Creamer, E. G., & Schoonenboom, J. (2018). Intermethod mixing as a gateway to methodological innovation. American Behavioral Scientist, 62(7), 879–886. https://doi.org/10.1177/00027642 18756917 Creswell, J., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research. Sage. Curry, L. A., O’Cathain, A., Clark, V. L. P., Aroni, R., Fetters, M., & Berg, D. (2012). The role of group dynamics in mixed methods health sciences research teams. Journal of Mixed Methods Research, 6(1), 5–20. https://doi.org/10.1177/ 1558689811416941 DeJonckheere, M., Lindquist-Grantz, R., Toraman, S., Haddad, K., & Vaughn, L. M. (2018). Intersection of mixed methods and community-based participatory research: a methodological review. Journal of Mixed Methods Research. https://doi.org/ 10.1177/1558689818778469 Deschepper, R., Six, S., Vandeweghe, N., De Couck, M., Gidron, Y., Depoorter, A.-M., & Bilsen, J. (2017). Linking numbers to perceptions and experiences: why we need transdisciplinary mixedmethods combining neurophysiological and qualitative data. Methodological Innovations, 10(2), 205979911770311. https://doi.org/10.1177/ 2059799117703119 Ellis, C., Adams, T. E., & Bochner, A. P. (2011). Autoethnography: an overview. Historical Social Research, 36(4), 273–290. https://doi.org/10.17169/fqs12.1.1589 Fam, D., Neuhauser, L., & Gibbs, P. (2018). Transdisciplinary theory, practice and education: the art of collaborative research and collective learning. In Transdisciplinary Theory, Practice and Education: The Art of Collaborative Research and Collective Learning. https://doi.org/10.1007/978-3-31993743-4 Fetters, M. D., & Freshwater, D. (2015). The 1 + 1 = 3 integration challenge. Journal of Mixed Methods Research, 9(2), 115–117. https://doi.org/10.1177/ 1558689815581222 Freshwater, D., & Cahill, J. (2013). Paradigms lost and paradigms regained. Journal of Mixed Methods Research, 7(1), 3–5. https://doi.org/10.1177/ 1558689812471276 Given, L. M. (Ed.) (2008). The Sage Encyclopedia of Qualitative Research Methods (Vols. 1-0). Sage. https://doi.org/10.4135/9781412963909 Greene, J. C. (2007). Mixed methods in social inquiry (1st ed.). Wiley. Greene, J. C. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2(1), 7–22. https://doi.org/ 10.1177/1558689807309969 Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational

525

Evaluation and Policy Analysis, 11(3), 255–274. https://doi.org/10.3102/01623737011003255 Hemmings, A., Beckett, G., Kennerly, S., & Yap, T. (2013). Building a community of research practice: Intragroup team social dynamics in interdisciplinary mixed methods. Journal of Mixed Methods Research, 7(3), 261–273. https://doi.org/10.1177/ 1558689813478468 Hesse-Biber, S. (2016). Doing interdisciplinary mixed methods health care research. Qualitative Health Research, 26(5), 649–658. https://doi. org/10.1177/1049732316634304 Hudd, S. S., & Bronson, E. F. (2007). Moving forward looking backward: An exercise in recursive thinking and writing. Teaching Sociology, 35(3), 264–273. https://doi.org/10.1177/0092055X0703500305 Jahn, T., Bergmann, M., & Keil, F. (2012). Transdisciplinarity: Between mainstreaming and marginalization. Ecological Economics, 79, 1–10. https:// doi.org/10.1016/j.ecolecon.2012.04.017 Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: a research paradigm whose time has come. Educational Researcher, 33(7), 14–26. https://doi.org/10.3102/0013189X033007014 Johnson, R.B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a Definition of Mixed Methods Research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/15586898 06298224 Knorr-Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge. Harvard University Press. Klein, J.T. (2018). Learning in Transdisciplinary Collaborations: A Conceptual Vocabulary. In: Fam, D., Neuhauser, L., Gibbs, P. (eds) Transdisciplinary Theory, Practice and Education. Springer, Cham. https://doi.org/10.1007/978-3-319-93743-4_2 Kuckartz, & Rädiker, S. (2019). Analyzing qualitative data with MAXQDA: Text, audio, and video. Springer. Legg, C. & Hookway, C., (2021). “Pragmatism,” The Stanford Encyclopedia of Philosophy, Edward N. Zalta (ed.). Online: plato.stanford.edu/archives/ sum2021/entries/pragmatism/. Lunde, Å., Heggen, K., & Strand, R. (2013). Knowledge and power: Exploring unproductive interplay between quantitative and qualitative researchers. Journal of Mixed Methods Research, 7(2), 197– 210. https://doi.org/10.1177/1558689812471087 Maxwell, J. A. (2016). Expanding the history and range of mixed methods research. Journal of Mixed Methods Research, 10(1), 12–27. https:// doi.org/10.1177/1558689815571132 Merriam-Webster Dictionary (2021). Recursive. Retrieved from: www.merriam-webster.com/dictionary/recursive Mertens, D. M. (2010). Transformative mixed methods research. Qualitative Inquiry, 16(6), 469–474. https://doi.org/10.1177/1077800410364612

526

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Mitchell, C., Cordell, D., & Fam, D. (2015). Beginning at the end: The outcome spaces framework to guide purposive transdisciplinary research. Futures, 65, 86–96. https://doi.org/10.1016/j.futures.2014. 10.007 Mitchell, P. H. (2005). What’s in a name? multidisciplinary, interdisciplinary, and transdisciplinary. Journal of Professional Nursing, 21(6), 332–334. https://doi.org/10.1016/j.profnurs.2005.10.009 Morgan, D. L. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48–76. https://doi.org/10.1177/2345678906292462 O’Cathain, A., Murphy, E., & Nicholl, J. (2008). Multidisciplinary, interdisciplinary, or dysfunctional? Team working in mixed-methods research. Qualitative Health Research, 18(11), 1574–1585. https://doi.org/10.1177/1049732308325535 Ojala, Selin, J., Partala, T., & Rossi, M. (2020). Virtual construction: interactive tools for collaboration in virtual reality. Advances in Information and Communication, 341–351. https://doi.org/10.1007/ 978-3-030-39442-4_26 Pawson, R., & Tilley, N. (1997). An introduction to scientific realist evaluation. London: Sage. Popa, F., & Guillermin, M. (2017). Reflexive methodological pluralism: the case of environmental valuation. Journal of Mixed Methods Research, 11(1), 19–35. https://doi.org/10.1177/1558689815610250 Popa, F., Guillermin, M., & Dedeurwaerdere, T. (2015). A pragmatist approach to transdisciplinarity in sustainability research: From complex systems

theory to reflexive science. Futures, 65, 45–56. https://doi.org/10.1016/j.futures.2014.02.002 Poth, C. (2018). Innovation in mixed methods research a practical guide to integrative thinking with complexity. Sage. Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. https://doi.org/10.1007/BF01405730 Rosenfield, P. L. (1992). The potential of transdisciplinary research for sustaining and extending linkages between the health and social sciences. Social Science and Medicine, 35(11), 1343–1357. https://doi.org/10.1016/0277-9536(92)90038-R Sanscartier, M. D. (2020). The craft attitude: Navigating mess in mixed methods research. Journal of Mixed Methods Research, 14(1), 47–62. https:// doi.org/10.1177/1558689818816248 Shannon-Baker, P. (2016). Making paradigms meaningful in mixed methods research. Journal of Mixed Methods Research, 10(4), 319–334. https:// doi.org/10.1177/1558689815575861 Shannon-Baker, P., & Edwards, C. (2018). The affordances and challenges to incorporating visual methods in mixed methods research. American Behavioral Scientist, 62(7), 935–955. https://doi. org/10.1177/0002764218772671 Teddlie, C., & Yu, F. (2007). Mixed methods sampling. Journal of Mixed Methods Research, 1(1), 77–100. https://doi.org/10.1177/1558689806292430 Watson, A. (2020). Methods braiding: A technique for arts-based and mixed-methods research. Sociological Research Online, 25(1), 66–83. https:// doi.org/10.1177/1360780419849437

36 Mapping Design Trends and Evolving Directions Using the Sage Handbook of Mixed Methods Research Design J o h n W. C r e s w e l l , C h e r y l N . P o t h and Peter Rawlins

MAPPING TO ADVANCE MIXED METHODS RESEARCH CONVERSATIONS In this final Section 6 chapter, we report the results of a mapping exercise to convey how the mixed methods design trends are represented in the Sage Handbook of Mixed Methods Research Design (subsequently referred to as the Handbook). We offer a commentary on four evolving directions and point to new as well as continuing ways the directions advance mixed methods research design conversations. Creswell has long used “mapping” as a useful pedagogical tool for helping mixed methods researchers make sense of the field and situating their contributions to the field (Creswell, 2009). Experience tells us the overviews that often accompany mapping descriptions are helpful for all, and especially for newcomers who describe increasing difficulty navigating the exponentially growing mixed methods researchspecific literature. We see the importance of such maps to guide future directions and provide a record of where the field of mixed methods research has been. In the second edition of The Sage Handbook of Mixed Methods in Social & Behavioral Research, Creswell (2010) described a need for mapping the field, saying:

It is time to reflect on this developing landscape and to map discussions about issues, priorities, and topics that have emerged. Such a mapping can provide a status report of the field of mixed methods, provide new scholars to the field of mixed methods with a general guide for positioning their studies within the mixed methods literature, and help encourage focused discussions among experienced researchers familiar with the literature on mixed methods. This chapter maps key developments in mixed methods research and suggests future issues that need to be addressed. (p. 46)

Researchers working in a rapidly changing field such as mixed methods research often face difficulties identifying specific ‘trees’ representing key developments and issues amidst the ‘forest’ of ever-growing literature. Pertinent to the Handbook focused on mixed methods research design is that even in his early mapping efforts, Creswell (2009) acknowledges that the proposed types of mixed methods designs are not complex enough to mirror actual practice. In so doing, he makes clear that the future of mixed methods design was, and remains to this day, yet evolving and unknown. Given the ongoing dialogues related to design, the time is right to inform further design practice advancements to better prepare researchers for the

528

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

dynamic realities they will encounter in their studies (Poth, 2018). The purpose of this chapter is to convey mixed methods design trends represented in the Handbook and offer our projection of four emerging directions for mixed methods research design. It is important for us to recognize, as Creswell has in previous mapping reports, that this chapter cannot be considered exhaustive, but rather is a dialogue bounded by the topics covered and threads of conversations by authors of the Handbook. Following this introduction of how mapping has and continues to contribute to advancing conversations within the field of mixed methods, we describe our collaborative approach and unique contributions to this mapping chapter for mixed methods research design. We discuss leveraging handbooks as indicators of design trends and situate the questions and issues raised by the Handbook among the unique features and key characteristics of four previous mixed methods research-focused handbooks. To frame our examination of the topic coverage by the Handbook, we map chapter topics with the 2016 Mixed Methods International Research Associaiton (MMIRA) Task Force Report (Mertens et al., 2016a, 2016b) and with Creswell’s (2022) MMIRA symposium cross-cutting Handbook themes. We conclude with

a commentary on the evolving directions represented by the Handbook and new as well as continuing ways the directions advance mixed methods research design conversations.

OUR COLLABORATIVE APPROACH AND CONTRIBUTIONS TO THIS CHAPTER In this chapter collaboration, we draw upon our differing backgrounds and involvement in the Handbook to generate insights that would not have otherwise been possible by any of us working independently. Collectively, we bring more than eight decades of qualitative, quantitative and mixed methods research experience, as well as extensive educational backgrounds as instructors, administrators and mentors. Central to this chapter is the potential for others to learn from both our collaborative process and outcomes. To that end, Figure 36.1 depicts three key contributors to our mapping process (i.e., extending symposium discussions, leveraging previous handbooks and revisiting 2016 Task Force projections) that led to our outcomes involving four evolving directions

Figure 36.1  Key contributors to our mapping process and outcomes

MAPPING DESIGN TRENDS AND EVOLVING DIRECTIONS USING THE SAGE HANDBOOK

for mixed methods design conversations. This chapter offers an example of a synergistic collaboration across oceans and time zones (with Creswell located in Hawaii and Japan, Poth in western Canada and Rawlins in New Zealand), aided by digital technologies, and supported by our active membership in the global mixed methods research community. We also acknowledge that our perspectives presented in this chapter are limited by our lived experiences and the experiences others have shared with us and cannot represent the full spectrum of diverse perspectives that are valued by our global mixed methods research community. This chapter, emerging from discussions that took place during a symposium focused on the Handbook at the 2022 MMIRA Global Conference, illustrates the value of conferences as platforms for disseminating ideas and launching new collaborations initially described by Mertens (2014). We discovered that our different roles in the development of the Handbook provided unique viewpoints for this conversation. As the Handbook Editor and author of the introductory and concluding chapters, Poth sought to involve scholars from around the world offering diverse viewpoints as chapter authors, section leads, and advisory board members for the Handbook (for a further description of the Handbook’s development, see Chapter 1, this volume). Creswell served as an International Advisory Board member and chapter author in Section 1 of the Handbook, as well as being the symposium discussant. As the co-lead of Section 6 and chapter author in Section 3, Rawlins has been involved with the Handbook throughout its development. Our lived experiences also contributed to our unique viewpoints and enriched our collaborative efforts. As a central and influential figure in the field of mixed methods research, Creswell has worn many hats including (but not limited to) being the cofounding editor of the premier journal in the field, Journal of Mixed Methods Research (JMMR; https://journals.sagepub.com/ home/mmr), a founding board member and then inaugural MMIRA president (https://mmira.org) (Mertens, 2014), and co-author of the best-selling textbook, now in its 3rd edition, Designing and Conducting Mixed Methods Research (Creswell & Plano Clark, 2007, 2011, 2018). His significant contributions to mixed methods capacity building is evident in his roles as cofounding director of the University of Michigan Mixed Methods Research programme and co-lead of the National Institutes of Health (NIH) best practices in mixed methods study group (NIH Office of Behavioral and Social Sciences, 2018). As an associate editor of the JMMR and having served on the founding board and then as fourth MMIRA President,

529

Poth is an established and active member of the global mixed methods research community. Her book, Innovation in Mixed Methods Research: Guiding Practices for Integrative Thinking with Complexity (2018, Sage), was inspired by the design practice dilemmas experienced in the field. In seeking to develop his university’s capacity within the mixed methods research field in 2015, Rawlins reached out to MMIRA and invited then-president, John Creswell and then-executive director, Burke Johnson, to lead a two-day national seminar. Rawlins was instrumental in establishing the Oceania Regional MMIRA chapter, co-chairing the inaugural Oceania MMIRA regional conference in Wellington, New Zealand in 2019, and the 2022 Global MMIRA conference. Together our sustained involvement in this Handbook and our lived experiences in the global mixed methods research community contributed to the rich discussions that ultimately led to this chapter and the discussion of the mixed methods research design trends as represented by the Handbook.

LEVERAGING MIXED METHODS HANDBOOKS FOR MAPPING THE “CURRENT STATE OF TRENDS” Handbooks serve important functions in the field of mixed methods research: to bring diverse perspectives from scholars around the world together and provide an opportunity to share and build upon those perspectives. This occurs over extended time when writing chapters for others to read and critique. Such events can capture the evolutions of practice advancements and serve as a key event. Recently, Molina-Azorin and Fetters (2022) described the publication of the first handbook focused on mixed methods in 2003 as an early key event for the field. To explore how the handbooks have and continue to capture the current state of design trends at the time of publication, we look to key characteristics across five mixed methods research-focused handbooks (see Table 36.1). Among the key patterns we see when comparing across the five handbooks is that, over time, the handbooks have generally increased in number of chapters, authors, and countries represented by authors. At the same time, we note the efforts to increase the breadth of countries involved as positive, we also recognize the necessity of further work to diversify the perspectives and cultural contexts represented in handbooks. This Handbook is unique in its focus on design and organizational structure using six

530

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 36.1  Key characteristics of previous mixed methods research handbooks Year/Title

Editors/Publisher

Number of chapters (organizational structures)/ Total pages

Number of authors/countries*

2003 Handbook of mixed methods in social & behavioral research 2010 Sage Handbook of mixed methods in social & behavioral research (2nd ed.). 2015 The Oxford handbook of multimethod and mixed methods research inquiry 2022 The Routledge handbook for advancing integration in mixed methods research 2023 Sage handbook of mixed methods research design

Tashakkori, T. & Teddlie, C. Sage

26 (in 4 sections) 768

42 6

Tashakkori, T. & Teddlie, C. Sage

31 (in 3 parts) 893

52 8

Hesse-Biber, S., & Johnson, 40 (in 5 parts) R.B. 737 Oxford

70 11

Hitchcock, J. H., & Onwuegbuzie, A. J. Routledge

34 (in 7 parts) 634

58 15

Poth, C. Sage

46 chapters (in 6 sections each with introductions and conclusions) 600

80 16

Note: *Each author’s country is represented by primary author affiliation in their biographies. Source: Author created.

sections, each led by editorial section leads, and its use of a community-sourcing approach. In the Handbook introductory chapter, Poth describes her role to initially identified the distinct topics for each section and possible authors during the proposal stage and then as a conductor supporting its development and performance. Poth also described the use of a community-sourcing approach to increase the real-world relevance of the Handbook’s discussion of innovative design practices. Common to all three of the Sage handbooks is the inclusion of mapping chapters, each with a different focus, that point to evolutions in the field. Whereas the inaugural Sage handbook (Tashakkori & Teddlie, 2003) focused on six issues and controversies, the second edition of the Sage handbook (Tashakkori & Teddlie, 2010) shifted to examining emerging pedagogy across three sections. The Oxford handbook (HesseBiber & Johnson, 2015) provided “a place for acknowledging points of disagreement, convergence, and sometimes contention as well as conciliatory views that exist within and between diverse multimethods and mixed methods communities” (p. xiii). The Routledge handbook (Hitchcock & Onwuegbuzie, 2022) focused on integration. The key issues raised by each of the six sections in this Handbook are summarized in Table 36.2.

REVISITING THE 2016 MMIRA TASK FORCE REPORT PROJECTIONS Creswell, in his role as founding MMIRA President, established a task force on the future of mixed methods and MMIRA and asked the seven international members to “cast our gaze to the future … identify fertile topics and challenges that members of this community may engage with the next five years (2016–2020)” (Mertens et  al., 2016a, p. 1). After brainstorming, drafting, and revising, the report identified five topics: • Definition, character and history of mixed methods. • Purposes, questions, design research and technological advances. • Social justice and the mixed methods researcher’s responsibility. • Teaching mixed methods research. • MMIRA, the profession, and professional development. To disseminate the report findings widely, the authors designed what they called “a kaleidoscopic look” into the future in a JMMR article, with the aim of engaging the mixed methods research community in a conversation about future

MAPPING DESIGN TRENDS AND EVOLVING DIRECTIONS USING THE SAGE HANDBOOK

531

Table 36.2  This Handbook’s six section titles, leads, key questions and unique issues Handbook section and title (Poth, 2023)

Section leads (primary country affiliation)

Key questions raised by each section

Unique issues raised by each section and examples

Section 1: Inspiring Diversity, and Innovations in Mixed Methods Design Section 2: The Craft of Mixed Methods Research Design

Sergi Fàbregues (Spain) and José Molina-Azorin (Spain)

What is mixed methods: a design or a methodology?

Judith Schoonenboom (Austria) and Sophia Johnson (USA)

How should we view mixed methods research?

Section 3: Expanding Design Approaches

Peggy Shannon-Baker (USA) How have mixed methods and Jessica DeCuir-Gunby designs expanded? (USA)

Section 4: Designing Innovative Integrations with Technology

Timothy Guetterman (USA)

What design approach should guide? (e.g., typology versus interconnected parts) What descriptor is useful? (e.g., a craft versus a disposition to uncertain, flexible, creative, emergent, ethical, and equitable design conditions) To what extent has the expansion been intentional? (e.g., priority-driven designs, international perspectives, and new types of complex designs) In what ways has technology been useful to advance integration? (e.g., software, visualizations, secondary data, gamebased applications) What is the nature of cultural influence? (e.g., periphery, central, specific research parts, funding) Where might we see innovation? (e.g.,visuals, grand societal challenges, translation of research into practice, interdisciplinary collaborations)

What innovations have emerged using technology and integration?

Section 5: Elsa Lucia Escalante Barrios How does culture influence Navigating Research (Colombia) and Elizabeth components of mixed Cultures in Mixed Creamer (USA) methods designs? Methods Research Design Section 6: Peter Rawlins (New Zealand) What is the future for Exploring Design Possibilities and Maggie Hartnett mixed methods research and Challenges for Mixed (New Zealand) designs? Methods Research

directions for mixed methods research (Mertens et  al., 2016b). In so doing, they offered implications related to four key themes that were inspired by the five topics described in the report: • • • •

Conceptual and methodological advances. Technology and big data. Preparation of mixed methods researchers. Complex social problems.

In a subsequent reflection also published in JMMR, Creswell (2016) provided his perspective drawing on his 35 years in the field. He described how he came to see the synergy of combining

quantitative and qualitative data that led to writing his early book, Research design: Quantitative and qualitative approaches (Creswell, 1994). Importantly, he also highlighted that in 2016, the field found itself in a period of design openness where it was “creatively enlarging [its] paradigmatic stances, especially through the transformative paradigm, and more recently, the eclectic pluralism perspective” (Creswell, 2016, p. 216). He pointed to the expansion of more advanced designs, the “new ways to visualize designs through graphical and pictorial displays” (Creswell, 2016, p. 217), and the use of teams in mixed methods research. Importantly, he also noted the lack of focus on integration given its

532

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

role as a key distinguishing feature of mixed methods research. This Handbook was not specifically designed to respond to the future directions identified in the MMIRA task force report. Our mapping of the chapters against the four key themes identified in the JMMR article extended Creswell’s (2022) initial findings that many of the chapters touched on the cross-cutting themes as symposium discussant. In Table 36.3, we present these findings not as exhaustive, but rather centred on the primary focus of the chapters when mapped against the themes from the task force report. In this Handbook, Chapter 32 seemed to fit within the two themes of technology and big data, and the preparation of mixed methods researchers, and Chapter 4 seemed to fit within the two themes of cyclical designs and complex social problems so these chapters were the only ones that were included across two themes. Our rationale for conducting this mapping exercise was, in part, to see if the conversation started by Mertens et  al. (2016b) and Creswell (2016) coincided with the directions the mixed methods research community had moved in over the last six years. In Chapter 5 (this volume), Niglas envisioned several fertile areas for the future development of mixed methods research related to mixed methods design. She launched her discussion of current issues: “[have] these important trends … been nurtured by the members of the mixed methods community and [will they] continue to be actual in the coming years?” (p. 60). Put simply, have the four themes identified in the 2016 MMIRA Task Force Report been productive areas of development? From this mapping exercise, we extrapolate that, although many of the Handbook authors were probably not writing with the 2016 MMIRA Task Force Report (Mertens et  al., 2016a) or JMMR article (Mertens et  al., 2016b) on their mind. Still, there is a close alignment between the areas they have been involved in and are now writing about and those four themes in the task force report. We suggest that this is not coincidental, but rather an indication that the task force report “got it right”. In Table 36.3 we have only mapped 19 of this Handbook’s 34 chapters (excluding section introductions, conclusions, this chapter, and the two chapters written by the Editor). It might also be noted that all the Section 1 Handbook chapters are included in Table 36.3, as well as the majority of chapters from Sections 2 and 6. See the next section for how the remaining chapters helped shape our discussion of four emerging directions for mixed methods research.

FOUR EVOLVING DIRECTIONS FOR MIXED METHODS RESEARCH DESIGN CONVERSATIONS You might now be asking, so what evolving directions does this Handbook point to? Creswell (2022) identified four cross-cutting themes in the Handbook that he anticipates as areas for further mixed methods research design practice evolutions: • Embracing emergent, flexible and uncertain designs. • Valuing international applications and cultural adaptations of designs. • Describing innovative technology design applications. • Addressing societal issues with design intersections. Each of these areas are discussed below, and in Table 36.4 we have used these areas as the framework on which to map the Handbook’s chapters. Again, this mapping exercise is not meant to be exhaustive, but rather to provide examples of the links we see with this Handbook’s chapters. In this mapping exercise, we see how the remaining 14 chapters of this Handbook’s 34 chapters relate to the cross-cutting themes identified by Creswell (2022). We draw your attention to the prevalence of Handbook chapters from Sections 3, 4 and 5. We suggest that this Handbook provides readers with opportunities to read about not only areas in which the task force had foreseen, but also new and emerging topics in need of further development.

Embracing Emergent, Flexible and Uncertain Designs The first cross-cutting theme involves opening up our discussions about mixed methods research design beyond those that can be predetermined and then conducted as planned. The timing of this Handbook’s development coincided with the global COVID-19 pandemic. That pandemic forced many changes to how we conducted our daily lives, particularly how we worked. The need to ‘pivot’ and adapt in uncertain times made us all the more aware of how plans can, and must, change, and may have influenced how the concept of “emergence” appeared throughout this Handbook. Concerning mixed methods research designs, this cross-cutting theme underscores the messiness of mixed methods research. The illustrative examples in this Handbook provided insights into how researchers navigated the

Transdisciplinary approaches Acceptance of mixed methods research Professional development Unique skills

Wicked problems New designs needed

Complex social problems

Secondary data Visualizations

Big data

Preparation of mixed methods researchers

Technology and big data

Multiple paradigms Cyclical designs

Conceptual and methodological advances

Integration Creative thinking Stakeholder involvement

Symposium discussant themes identified by Creswell (2022)

Challenges and opportunities identified by Mertens et al., (2016b)

Chapter 6 (Section 1) on the role of methodological paradigms for dialogic knowledge production Chapter 3 (Section 1) on a historical perspective of designs Merten’s section 1 chapter on transformative designs considered to be cyclical Chapter 10 (Section 2) on data integration as a form of integrated mixed analysis Chapter 9 (Section 2) on using a tree metaphor as a sampling meta-framework in mixed methods research Chapter 17 (Section 3) using participatory methods in randomised controlled trials of complex interventions Chapter 29 (Section 5) on community-based participatory approaches to advance health equity Chapter 11 (Section 2) on ethical issues in an era of big data Chapter 23 (Section 4) on text mining Chapter 25 (Section 4) on using existing data Chapter 8 (Section 2) on creatively visualizing the interaction between methods and inferences Chapter 32 (Section 6) on using visuals to teach mixed methods research and the analysis of visually oriented data from sources such as social media Chapter 35 (Section 6) on opportunities and challenges for transdisciplinary mixed methods research Chapter 5 (Section 1) on developments in mixed methods designs Chapter 2 (Section 1) on revisiting mixed methods research designs Chapter 32 (Section 6 ) on using visuals to teach mixed methods Chapter 20 (Section 3) on instrument development and legitimation Chapter 26 (Section 4) on novel case-oriented merged analysis Chapter 33 (Section 6) on responsible mixed methods research for tackling grand challenges for the betterment of society Chapter 4 (Section 1) on transformative designs to further social, economic, and environmental justice Chapter 16 (Section 3) on prioritizing cultural responsiveness in mixed methods research and team science with underrepresented communities

Topics covered in the Handbook and example chapters and their sections

Table 36.3  Mapping projections from Mertens et al. (2016b), Creswell’s (2022) symposium themes, topics, and example chapters from this Handbook

MAPPING DESIGN TRENDS AND EVOLVING DIRECTIONS USING THE SAGE HANDBOOK 533

534

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Table 36.4  Mapping Handbook chapters onto the evolving directions for mixed methods research design Evolving directions identified by Creswell (2022)

Remaining 14 Handbook (Poth, 2023) chapters and topics not mentioned in Table 36.3

Embracing Emergent, Flexible and Uncertain Chapter 7 (Section 2) on embracing emergence Designs Chapter 12 (Section 2) on building the logic for an integrated methodology Valuing International Applications and Chapter 14 (Section 3) on Indigenous cultural values instrument Cultural Adaptations of Designs development in Iran Chapter 15 (Section 3) about kaupapa Ma¯ ori research principles Chapter 27 (Section 5) on culturally responsive mixed methods inquiry Chapter 28 (Section 5) integrating a four-step Japanese cultural narrative framework Chapter 30 (Section 5) on cultural diversity in implementation designs in China Chapter 31 (Section 5) examination of the influences of Spanish research culture in systematic observation with mixed methods Describing New Technology Design Chapter 22 (Section 4) on using software in new ways Applications Chapter 24 (Section 4) on game-based research integrations Addressing Societal Issues with Design Chapter 21 (Section 3) on mixed method grounded theory Intersections Chapter 13 (Section 3) on intersectionality-informed mixed methods research Chapter 18 (Section 3) on mixed methods phenomenological approach Chapter 19 (Section 3) on mixed methods and case study research Chapter 34 (Section 6) on knowledge translation

messiness in their mixed methods research designs. As such, we imagine a future where researchers adopt a more emergent approach to their planning and conducting of studies, and depart from the assumptions that designs can be predetermined and processes involved can be considered to be linear. Creswell (2022) argued that thinking about mixed methods research as a creative approach represented an important area advanced in this Handbook.

Valuing International Applications and Cultural Adaptations of Designs The second cross-cutting theme identified by Creswell in the Handbook involves enhancing the visibility of mixed methods designs that illustrate international design applications and describe cultural context adaptations. While societal issues discussed in the task force report addressed the need for cultural responsiveness (Mertens et  al., 2016a, 2016b), Creswell argued for further valuing of cultural adaptations of mixed methods designs. We suggest that a focus on cultural adaptations provides an opportunity for all researchers to engage in cultural reflection and make explicit how our designs are continually influenced by

what the researchers bring, as well as by the many contexts in which our studies take place. By design, this Handbook sought to have authors from around the globe contribute diverse illustrative examples. The inclusive orientation of this Handbook may have created space for authors to explore how mixed methods designs are adapted to suit a particular cultural context, and to discuss what more established mixed methods designs can learn from— for example, indigenous values and ways of researching. This Handbook reports studies representative of various geography, populations and cultures departing from the familiar for many readers. As such, it provides a useful opportunity for mixed methods researchers from around the world to learn from culturally diverse studies and incorporate aspects into their own design. Additionally, this cross-cutting theme signals that international applications and cultural adaptation of designs is a significant area of future development in the field of mixed methods research.

Describing Innovative Technology Designs Applications Thirdly, Creswell identified the description of innovative technology application designs as

MAPPING DESIGN TRENDS AND EVOLVING DIRECTIONS USING THE SAGE HANDBOOK

another major cross-cutting theme, which indicates a broadening use of technology in all aspects of the mixed methods research design process. We have experienced the ubiquitous use of technology during the COVID-19 pandemic, which has significantly changed how we interact with others when conducting mixed methods research. Technology now influences the range of activities we engage in, from how teams meet to how we collect, analyze, and represent data and even disseminate our insights with colleagues. Creswell argued that the way that mixed methods research is now responding to new technological advances underscores a new roadmap. As such, mixed methods researchers should be prepared to use technology in innovative ways to communicate with others, analyze new sources and types of data such as the increased use of visual data on social media, create novel designs drawing upon big data, and visually represent integration evidence, to name a few. We argue that researchers should have the opportunity to be exposed to and develop skills that allow them to begin exploring the limitless possibilities that technology affords. We note that many of the section leads for the Handbook represent those who are technology-aware and savvy in their practice, evidenced by their own use of technology. We wonder if this allowed them to have a greater awareness of the technology discussed in their chapters and encouraged technology to become more visible in their descriptions of their section during the MMIRA symposium. The ubiquitous nature of technology can also be observed in the development of this Handbook and the MMIRA online global conference, both fully mediated through technology.

Addressing Societal Issues with Design Intersections The final area identified as a cross-cutting theme is an extension of ideas that were reflected in the earlier work of the task force project about the role that new mixed methods research design interactions and practices play in addressing societal issues. The first three years of the pandemic were important in raising awareness of global inequalities and the complexities involved in pressing societal issues. Importantly, references to wicked and complex societal problems and grand challenges are front and centre in many of the Handbook’s chapters. Also highlighted are research teams, new combinations of designs with mixed methods research and stakeholder-involved approaches as optimal and necessary configurations for addressing those wicked problems and

535

grand societal challenges. A complementary area of development is how we assess the impact of mixed methods research. Its impact is no longer sufficient to simply publish peer reviewed articles and rely on the number of citations. Instead, several chapters discuss the need for knowledge translation and stakeholder involvement to bring about important practical impacts. Such a change in how the value of research is conceptualized will require a paradigm shift. It is essential that researchers gain early and continued exposure over their careers as well as opportunities to develop these specific skills.

IMPLICATIONS OF THE HANDBOOK FOR FUTURE MIXED METHODS DESIGN CONVERSATIONS Mapping the chapters of the Handbook onto the projections from the work of the 2016 task force (Mertens et al., 2016a, 2016b) provides important insights into how the field has developed in ways that were anticipated and have evolved as well as to identify new emerging directions. This work offers compelling evidence of the evolutions of mixed methods research practices in important design areas such as technology applications and culturally responsive practices. While we can assume that practice evolutions, such as the movement towards more emergent and flexible designs, have been influenced and informed by the rapidly changeable and unpredictable global circumstances, it is impossible to predict the extent to which or why evolutions will occur. As we gaze into the unknown future, we posit the essential role for access to and engaging in continued professional learning to keep pace with the almost certain practice advancements to come. Preparing future generations of mixed methods researchers has long been an important topic of conversation and debate (e.g., Creswell et  al., 2003; Christ, 2010; Poth & Munce, 2020; Chapter 32, this volume), yet it mostly focused on initial training rather than launching lifelong learning. Perhaps this emerging conversation is yet another indicator of acceptance and maturity for the field of mixed methods that comes with its increasing use reported by others (e.g., Chapter 2, this volume; Fetters & Molina-Azorin, 2021). We hope researchers will continue to grapple with important questions related to what are the knowledge, skills, and attitudes that are essential for mixed methods research and what are the promising approaches for guiding effective teaching and lifelong learning. We anticipate that how we build and assess evidence

536

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

of mixed methods research-specific capacity will continue to dominate the conversation for the next decade. In this chapter, we offered our projections of areas in need of continuing professional learning through our discussion of four emerging directions. In brief, mixed methods researchers should be prepared to navigate emergent and flexible designs, value culturally diverse designs, embracing technology design applications, and work with others for the betterment of society through mixed methods designs. As a result, mixed methods research learning opportunities need to be designed to offer opportunities for gaining knowledge about as well as developing the requisite design skills of the future. In writing this chapter, we reflected upon the enduring relevance of the projections in the 2016 MMIRA Task Force Report (Mertens et al., 2016a), subsequent JMMR article (Mertens et  al., 2016b), and Creswell’s (2022) reflection in the light of the Handbook. We anticipate that we will see concerted efforts during the next five years of culturally focused design of mixed methods research. We conclude with a caveat, in a similar spirit to what Creswell expressed in his 2009 editorial, that we recognize that a mapping exercise can be interpreted as an attempt to fix the field and provide a template to which new components must be assimilated. It is our hope that this chapter stimulates a conversation rather than an attempt to impose determinacy which we believe would be both impossible and undesirable to attempt. We look forward to a future mapping exercise to gauge the future relevance of the emerging directions discussed in this chapter and the insightful Handbook chapters for readers to discover!

WHAT TO READ NEXT Mertens, D. M., Bazeley, P., Bowleg, L., Fielding, N. G., Maxwell, J. A., Molina-Azorin, J. F., & Niglas, K. (2016). Expanding thinking through a kaleidoscopic look into the future: Implications of the Mixed Methods International Research Association’s task force report on the future of mixed methods research. Journal of Mixed Methods Research, 10(3), 221–227. https://doi.org/10.1177/ 1558689816649719

As the future projections in this article serve as the framework on which the Handbook chapters are mapped, this is key reading for understanding where the field has come during the past almost decade.

Creswell, J. (2009). Editorial: Mapping the field of mixed methods research. Journal of Mixed Methods Research, 3, 95-108. https://doi.org/10.1177/ 1558689808330883

As an initial description of mapping the field of mixed methods research, this chapter serves to illustrate the usefulness of the type of exercise undertaken in this chapter. Chapter 2, this volume.

As the initial chapter for Section 1 of this Handbook, John Creswell and Vicki Plano-Clark provide an essential historical overview of mixed methods research designs covering a variety of topics ranging from evolutions in definitions of mixed methods research and the names and elements of designs to advancements in diagrams and integration.

REFERENCES Christ, T. (2010). Teaching mixed methods and action research: Pedagogical, practical, and evaluative considerations. In The Sage Handbook of Mixed Methods in Social & Behavioral Research (2nd ed., pp. 643–676). Sage. Creswell, J. W. (1994). Research design: Qualitative and quantitative approaches. Sage. Creswell, J. (2009). Editorial: Mapping the field of mixed methods research. Journal of Mixed Methods Research, 3, 95-108. https://doi.org/10.1177/ 1558689808330883 Creswell, J. (2010). Mapping the developing landscape of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), The Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 45–68). Sage. Creswell, J. W. (2011). Controversies in mixed methods research. In N. K. Denzin and Y. S. Lincoln (Eds). The Sage Handbook of Qualitative Research (4th ed., pp. 269–284). Sage. Creswell, J. W. (2016). Reflections on the MMIRA the future of mixed methods task force report. Journal of Mixed Methods Research, 10(3), 215–219. https://doi-org.login.ezproxy.library.ualberta.ca/ 10.1177/1558689816650298 Creswell, J. W. (2022, August 4). In C. Poth (Chair), Exploring current and future mixed methods research design possibilities: A discussion of the forthcoming Sage Handbook of Mixed Methods Design [Symposium discussion]. 2022 MMIRA Global Conference. Virtual. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Sage.

MAPPING DESIGN TRENDS AND EVOLVING DIRECTIONS USING THE SAGE HANDBOOK

Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Sage. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage. Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003a). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209–240). Sage. Creswell, J., & Poth, C. (2017). Qualitative inquiry & research design (4th ed.). Sage. Creswell, J., Tashakkori, A., Jensen, K., & Shapley, K. L. (2003b). Teaching mixed methods research: Practices, dilemmas, and challenges. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 619–637). Sage. Fetters, M. D., & Molina-Azorin, J. F. (2021). Guidance on using mixed methods from diverse international organizations in the behavioral, social, fundamental and health sciences. Journal of Mixed Methods Research, 15(4), 470–484. https://doi. org/10.1177%2F15586898211049629. Hesse-Biber, S., & Johnson, R.B. (Eds.). (2015). The Oxford handbook of multimethod and mixed methods research inquiry. Oxford. Hitchcock, J. H., & Onwuegbuzie, A. J. (Eds.). (2022). The Routledge handbook for advancing integration in mixed methods research. Routledge. Mertens, D. M. (2014). A momentous development in mixed methods research. Journal of Mixed Methods Research, 8, 3-5. https://doi.org/10.1177/ 1558689813518230 Mertens, D. M., Bazeley, P., Bowleg, L., Fielding, N. G., Maxwell, J. A., Molina-Azorin, J. F., & Niglas, K. (2016a). The future of mixed methods: A five year

537

projection to 2020. Retrieved from http://mmira. wildapricot.org/ Mertens, D. M., Bazeley, P., Bowleg, L., Fielding, N. G., Maxwell, J. A., Molina-Azorin, J. F., & Niglas, K. (2016b). Expanding thinking through a kaleidoscopic look into the future: Implications of the Mixed Methods International Research Association’s task force report on the future of mixed methods research. Journal of Mixed Methods Research, 10(3), 221– 227. https://doi.org/10.1177/1558689816649719 Molina-Azorin, J. F., & Fetters, M. D. (2022). Books on mixed methods research: A window on the growth in number and diversity. Journal of Mixed Methods Research, 16(1), 8–16. https://journals.sagepub. com/doi/pdf/10.1177/15586898211068208 NIH Office of Behavioral and Social Sciences. (2018). Best practices for mixed methods research in the health sciences (2nd ed). Retrieved from: www. obssr.od.nih.gov/wp-content/uploads/2018/01/ Best-Practices-for-Mixed-Methods-Research-inthe-Health-Sciences-2018-01-25.pdf Poth, C. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Poth, C. (2021). Research ethics. Sage. Poth, C., & Munce, S. (2020). Commentary— Preparing today’s researchers for a yet unknown tomorrow: Promising practices for a developmental mentoring approach to mixed methods research learning, International Journal of Multiple Research Approaches, 12. 56-64. https://doi.org/10.29034/ ijmra.v12n1commentary. Tashakkori, T. & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social & behavioral research. Sage. Tashakkori, T. & Teddlie, C. (Eds.). (2010). Sage Handbook of mixed methods in social & behavioral research (2nd ed). Sage.

Where to Next in Exploring Possibilities and Challenges for Mixed Methods Research for the Future? Section 6 Conclusions Peter Rawlins and Maggie Harnett

In this chapter we offer our concluding thoughts about the chapters in Section 6. We do this by looking at the five chapters through the lenses of the three key threads identified in the Introduction— namely, evidence, people and technology. While these threads are discussed separately, we acknowledge that they are interconnected. In the process of editing Section 6, we observed that the authors found it challenging to project into the future beyond five to ten years. While digital technologies was an identified thread, we argue that they will play a more significant role than explored by the authors, especially big data and artificial intelligence (AI) (Cyber security intelligence, 2020). The first of these threads looked at evidence and asked the question: What might constitute credible evidence in the future? We advance that two of our chapters addressed this theme, one directly (Chapter 32, Shannon-Baker) and one indirectly (Chapter 33, Molina-Azorin and Fetters). One of the challenging aspects of this thread is unpacking what is meant by “evidence”? Is the evidence the data we collect? Is it the findings that arise from our research? Or is it evidence of the impact that our research has on broader societal challenges? We argue that it is all of these things. Looking first at data as evidence, we ask the question, what new sources of data are we seeing

and what might be the implications of using these sources. In their chapter, Shannon-Baker identifies how social media is increasingly becoming a valid data source. In particular, Shannon-Baker identifies how visual data such as emojis, memes, gifs, videos and photographs are increasingly common. The increasing use of visuals as data brings with it the challenges of how to effectively analyse large quantities of data that social media generates. Shannon-Baker suggests that there will need to be developments in data analysis programmes, such as adopting artificial intelligence (AI) as a primary means of analysis. We agree with Shannon-Baker who argues that the use of social media data raises important questions about data sovereignty, ethics and informed consent. Similar questions around data sovereignty, ethics and informed consent should also be asked about big data sources. Big data is quickly becoming central to research. Data is increasingly being gathered about many aspects of our everyday lives, often without our being aware of it. For example, data collected about public transport use, supermarket purchases, online browsing habits and health data collected via wearable tech such as smartwatches and fitness trackers. This big data can potentially be mined to solve real world societal problems. Many of these databases are vast

SECTION 6 CONCLUSIONS

and can represent people from a wide spectrum of socioeconomic, cultural and/or geographical contexts. Working with such large datasets can allow more sophisticated analysis, potentially increasing the validity and generalisabilty of research findings. But ethical questions about who owns this data and who determines how it can be used need to be addressed. That said, the idea of being able to conduct research into society’s larger challenges without having to collect all of the data has huge potential. Indeed, the implications for all aspects of research could be far-reaching. Chapter 36, Creswell et  al. highlighted that the area of technology and big data had been identified in the 2016 MMIRA task force report (Mertens et  al., 2016) and they also identify the need for innovative technology application in research designs as one of their four emerging areas in MMR. Readers interested in this may also like to refer to Chapter 11 (this volume) on the ethical issues of researching with big data. If we next consider the notion of “findings” as evidence of research, we ask the question, what are the potential changes to the way we see the outcomes of research communicated and will this result in a broader research audience? ShannonBaker identifies the increased use of visual representations of findings in research reports and publications. For the near future, this is likely to be predominantly in print media but, with the increasing use of online reporting we can expect to see things like interactive tables, graphs, infographics and other visual media that can be easily shared on a range of online platforms, promoting interaction with, and use of, research findings. Perhaps we might see visual representations that continually update as new data becomes available? Lastly in the evidence thread, we consider the term “evidence” more broadly and look at the evidence that our research is being used to make a difference and help to address broader societal problems. In other words, what evidence do we have of the impact of our research? Molina-Azorin and Fetters contend that, too often, researchers examine topics that are of interest to them, but are not relevant to practitioners and society. The authors argue that the future of mixed methods research, is dependent on its usefulness to solving grand societal challenges. In order to provide evidence that research is making a difference, it should lead to actionable outcomes for practitioners, stakeholders and community members. Accordingly, the authors argue that researchers should move away from simply using the number of publications and citations as a measure of “impact” towards a model where impact is judged

539

by how research is used by practitioners and society to address what they describe as “grand challenges”. In this way, scholars can improve not only their academic impact, but also their practical and social impact through use of mixed methods research. Similarly, Creswell et al. argue that addressing societal issues with new design intersection and practices is one of the four emerging directions they identified when considering the chapters in this Handbook. The people thread explored capacity building within communities and teams. At the centre of much of the discussion was the notion of collaboration. In the chapter by Archibald, she identifies that bringing people together in a collaborative environment is the foundation of transdisciplinary mixed methods research. In addition to the integration of data and methods, she highlights the human components of integration as being integral to the success of mixed methods research. She continues that the merging of disciplines through the creation of transdisciplinary teams, particularly those emphasising other forms of knowledge and ways of being, has the potential to provide insights and alternative ways of approaching problems. At times, collaboration involves the bringing together of research teams, but it also involves collaboration with community and other stakeholders. For example, Chapter 34, Ivankova et al. stress that the development of collaborative partnerships with key stakeholders and community members, and being cognisant of community priorities, are key factors when designing and conducting mixed methods studies for knowledge translation. Similarly, Chapter 33, Molina-Azorin and Fetters discuss the importance of collaboration and argue that there should be an interactive relationship between the researcher and the relevant stakeholders. In this model, community members, stakeholders and practitioners should be fully involved as research partners, engaging in shared decision making and co-researching with scholars. In their argument, the co-creation of knowledge through collaborative practices with stakeholders is an important component of responsible and transformative research. We argue here that the concept of embracing emergent, flexible and uncertain designs identified by Creswell et  al. as a future direction for MMR is consistent with a collaborative interactive research teams approach. It is likely that, in such teams, the research process is less likely to be fixed and linear, and more likely to adapt during the research process. Readers interested in research designs that further social, ethics and environmental justice may also want to read Chapter 4 (this volume).

540

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

In identifying the importance of collaboration for the future of mixed methods research, the section authors also identify some areas for consideration when broadening research teams. In Chapter 35, Archibald identifies the importance of deliberate planning of team structure and organisation for effective collaborative teams. Ivankova et al. discuss how the design of effective mixed methods study teams should be cognisant of the implications of personal, interpersonal and organisational factors. Ivankova et  al. highlight that identifying collaborators with the necessary personal characteristics to support mixed methods translational research is important but could be challenging. They advocate for a distributed leadership approach combining research team leaders with discipline-specific expertise who can retain tactical decision-making, and an executive leadership branch of stakeholders and researchers to manage the overarching strategic directions of the research. While collaboration in bigger and more complex teams was explored in depth, notions of larger collectives or networks of p­ eople ­taking part in the research were missing from the ­chapters in our section. Possibilities such as public participation in research by volunteer citizens to help solve real world problems (aka citizen science) were not considered. One thing that we have learned during the ongoing COVID 19 pandemic and the need to “work from home” are new ways to collaborate. While the use of collaborative technologies have been increasing in recent times, the pandemic has accelerated their use and acceptance. In times when it was difficult if not impossible for people to travel and gather together, we have had to develop new skills in the use of conferencing tools such as Zoom, Teams, etc., to meet together virtually, and collaborative tools such as Google docs, One Drive etc., and Miro. This was an area where the people thread and the technology thread intertwined, and will continue to do so in the future. In her chapter, Archibald identifies that communication and technological considerations will be instrumental in supporting current and future collaborative initiatives. Certainly, there is significant potential for technology to help people come together in teams and organise and cohere in new ways. Archibald speculates whether we will see developments such as virtual reality-based collaboration platforms to complement existing video conferencing platforms. Existing platforms tend to reinforce the “one person speaking at a time model” of conferencing whereas face to face meetings tend to have more interplay between participants with multiple participants speaking at the same time. Virtual reality platforms could move us closer to this model.

The technology thread examines the potential for digital tools to impact the way that research is conceptualised and conducted in the future. In this Conclusion, we have already talked about how technology can support collaborative research activities. But what other roles might technology play in helping us develop mixed methods research in future focused directions? The reader might like to refer back to Section 4 chapters which are centred on the impact of technology on mixed methods research. In their chapter, Shannon-Baker makes links between the technology thread and the people thread by convincingly arguing for the development of researcher capacity through the use of visuals, whether this be through researchers developing research diagrams to help them conceptualise research designs, or the use of joint displays to report findings from their research projects. Certainly, hybrid teaching models with on-line, F2F, synchronous and asynchronous teaching approaches are becoming more common with technology as the enabler. Creswell et al. reported that the preparation of mixed methods researchers was an area of future development identified in the 2016 MMIRA task force report (Mertens et al., 2016). As discussed earlier, with the publication of research transforming into more digitally based mediums, the use of visuals, such as interactive research diagrams, joint displays, augmented and virtual reality, will become increasingly common. These developments may fundamentally alter the structure of what it means to publish research. Considering mixed methods research more broadly, we note that the adoption of new digital technologies, particularly AI, are already fundamentally changing academic research. For example, AI is currently being used to assist with the research peer review processes by identifying plagiarism (Cyber security intelligence, 2020). The next step is likely to entail the research process itself with AI playing an important role in data collection, analysis, synthesis of findings, drawing conclusions, and the representation of the outcomes of research. Additionally, an increased focus on applied research as industry and philanthropic funding becomes more dominant may mean that the open access and dissemination of research findings may be a condition of research funding. There remains a high degree of uncertainty as we start to emerge from the grips of the COVID-19 pandemic. This is true for the way we live, the way we work and particularly the way we conduct research. There are, however, some clear directions for MMR identified through the chapters in this section. While they are not exhaustive, these directions are indicative of the changes MMR will

SECTION 6 CONCLUSIONS

need to make if it is to adapt to researching in a contemporary world.

REFERENCES Cyber security intelligence. (2020). Using artificial intelligence in academic research. Cyber security

541

intelligence. www.cybersecurityintelligence.com/ blog/using-artificial-intelligence-in-academicresearch-5124.html Mertens, D. M., Bazeley, P., Bowleg, L., Fielding, N. G., Maxwell, J. A., Molina-Azorin, J. F., & Niglas, K. (2016). The future of mixed methods: A five year projection to 2020. Retrieved from http://mmira. wildapricot.org/

37 An Emerging and Exciting Future for Mixed Methods Research Design: Handbook Conclusions Cheryl N. Poth

Throughout the development of this Handbook, I wondered how I might conclude this journey. My aim as Handbook Editor was to offer a global perspective of how researchers are leveraging the dilemmas and opportunities for mixed methods research designs with the aim to inspire mixed methods research design innovations (see also Chapter 1). This work has been influenced by thought-provoking conversations with, and my reading of insightful literature by, many members of the global mixed methods research community. As noted throughout this Handbook, mixed methods design holds unique potential for planning, conducting and communicating innovative processes, as well as for generating novel outcomes that have been previously inaccessible by either qualitative or quantitative approaches alone. The established and emerging innovators included in this Handbook break new ground in their chapter descriptions of the oft hidden influences on their mixed methods research designs. I thank all the Handbook contributors for inspiring new design conversations and practices. As a collection, the chapters speak to several questions introduced in Chapter 1 as having inspired the development of the Handbook: • What ought to be the scope of mixed methods research design?

• What mixed methods research design perspectives would benefit others to learn from and advance the field? • What processes and outcomes ought to be involved in future-forward mixed methods research design practices? • What recent practice advances ought to be incorporated into the design of future-forward mixed methods research? Together, the chapters guide our mixed methods research design conversations and practices in ways that are both expected and surprising. Many contributors to this Handbook promote a futureforward approach to the design of mixed methods research centred on creativity and openness. It is interesting that similar calls were made previously by the MMIRA Task Force report authors (Mertens et  al., 2016). In concluding this Handbook, I speculate that creative and open mixed methods research design thinking, conversations and practices are vital preparation for a yet unknown future. My intention here is to build on what contributors have done in this Handbook by speculating on the emerging and exciting design future for the global mixed methods research community. In discussing four design topics, I weave my

HANDBOOK CONCLUSIONS

own perspectives with ideas alluded to in the Handbook, illustrating some evolving landscapes of design terminology, illuminating many contextual influences on design practices, representing diverse design perspectives and assimilating practice evolutions in design education. I speculate about the challenges likely to be encountered for each topic and suggest actions aimed at moving the field forward. I close with a call for creative thinking and open conversations about mixed methods research design and education aimed at addressing current societal dilemmas and inspiring design innovations. I offer some concluding words to help realize the future-forward mixed methods research design innovations discussed in this Handbook.

ILLUSTRATING SOME EVOLVING LANDSCAPES OF MIXED METHODS RESEARCH DESIGN TERMINOLOGY How we define and use terminology has had implications for mixed methods research design. I predict evolutions in design terminology will occur alongside the exponential uptake of mixed methods research around the globe. Section 1 of this Handbook relates the evolving design dialogues from authors (see Chapters 2–6) who have experienced and contributed to the many crossroads of mixed methods research design practices. I imagine a future where design typologies expand at an unprecedented rate and are influenced by emerging literature such as the design naming practices advanced by Michael Fetters (2022). I anticipate that researchers will be challenged to keep pace with the rapid evolutions in design terminology and naming practices. Given the newness of Fetters’s editorial in the Journal of Mixed Methods Research at the time of publishing this Handbook, I can only speculate on its uptake and practical impact for achieving design naming conventions in future mixed methods research. I offer evidence of my own in-progress terminology evolutions in thinking about and defining the requisite integration in mixed methods research design. In discussing integration as the distinguishing feature of mixed methods research and thus of its design in the Handbook’s Introduction (see Chapter 1), I allude to a possibility of moving beyond dichotomizing data as qualitative or quantitative. This evolution is noteworthy because until recently, I had been steadfast in defining the requisite integration in mixed methods research

543

as necessitating both quantitative and qualitative data (e.g., Onwuegbuzie & Poth, 2016; Poth, 2018, 2020). In this evolution, I join Handbook authors (see Chapter 19) as well as others (e.g., Bazeley, 2018; Bergman, 2008; Mason, 2006; Pearce, 2015; Song et al., 2010) in calls for transcending what has been called a counterproductive “qualitative–quantitative divide”. It is not clear to me yet how this potential evolution in terminology will unfold in terms of timing or outcome. This experience leads me to call attention to the need for self-monitoring evolutions in our thinking and for tracking evolutions in key design terminology in emerging literature because these evolutions are likely to have practical yet unpredictable consequences. In calling attention to monitoring evolving landscapes of mixed methods research design terminology and its potential implications for the field of mixed methods research, I draw upon the impactful editorial by Fetters and colleagues (2021) and specifically their advocacy for acting upon the problem of racializing research rhetoric. These authors define racializing research rhetoric as “written and spoken language of research communities that reifies or perpetuates racism and racist systems of power or obscures the role of racism in the shaping of health and social inequities” (p. 8). I found their discussion of the term “segregated” as a descriptor within a mixed methods research design to be particularly of use when scrutinizing my own work and in informing my proactive approach in my work with chapter authors as editor of this Handbook. Similar to the authors of the editorial, I humbly acknowledge that I am not an expert in scientific racism or anti-racist scholarship, yet I view my responsibility as Editor of this Handbook as being to raise awareness of, take actions to address, and model an openness for scrutiny with the aim of lessening the presence of racialized research rhetoric in this Handbook. In this work, I found authors to be receptive. This is not to say that I was entirely successful, and I acknowledge the need for further learning on my part to recognize, react appropriately and remove the racism that infiltrates everyday life and writing. Please be brave and let me know what I have missed in this Handbook to further my learning. I imagine a future where mixed methods researchers define and use globally relevant and appropriate design terminology that avoids perpetuating structural racism in their proposals and publications. I call upon each of us to action this commitment in our mixed methods research work and to be open to practices that keep pace with the evolving landscapes of design terminology.

544

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

ILLUMINATING MANY CONTEXTUAL INFLUENCES ON MIXED METHODS RESEARCH DESIGN PRACTICES As a collection, the 35 chapters detail mixed methods research design contexts across six continents. Representing more than 50 unique study contexts poignantly illustrates a wide range of contextual influences on mixed methods research designs. That the design of mixed methods research is subject to changeable contextual influences that can create messy and uncertain conditions is well known (Plano Clark & Ivankova, 2016; Poth, 2018). Yet, the practice of providing detailed descriptions of the dynamic contextual influences remains less common than one might expect. To begin to address this gap and to provide guiding examples, this Handbook had, as an aim, for chapters to provide comprehensive descriptions of the key contextual influences shaping a study such as the participants, sites and researchers, as well as their surrounding environments (Poth, 2018). In published accounts of mixed methods research, comprehensive descriptions provide access to design details such as what was done, how it was done and why it was done that way. By applying different paradigm frames in her description of published mixed methods research accounts, Mertens (2023) provides novel practical guidance to help researchers clarify the assumptions that guide their mixed methods research design decisions. The design of mixed methods research rarely occurs as planned; much of the uncertainty can be attributed to the dynamic contextual influences. The range of possible contextual influences is staggering, for example, from cultural and social norms to historical and economical settings. Paying attention to the many sources of contextual influences on the design of mixed methods research is paramount, as is recognizing that these contexts are dynamic and those involved are influencing as well as being influenced by the changeable contexts. Sanscartier (2020) described a craft attitude as helpful for acknowledging and engaging with the messiness inherent in mixed methods research. Section 2 (see Chapters 7–12) illustrates the craft attitude and provides practical design guidance for mixed methods researchers through descriptions of navigating uncertainty. Whereas some chapters describe the dynamic nature of the research contexts as necessitating emergent approaches to design (Chapters 7 and 8), others make the case for adaptive approaches (Chapters 10 and 12). Descriptions can inspire thoughtful reflections, helping researchers recognize how their

backgrounds and experiences shape their design decisions, and make such understandings accessible for others to learn from and emulate. In developing this Handbook, we were intentional in seeking illustrations of international design applications and those describing cultural context adaptations. Specific to Section 3 (see Chapters 13–21), we sought to expand our understanding of the influences of cultural contexts and intersections of other designs with mixed methods (see also Section 3 Introduction). Specific to Section 5 (see Chapters 27–31), we asked researchers to describe their research contexts and comment on their potential influences on their mixed methods research designs (see also Section 5 Introduction). We had difficulty locating guiding examples and did our best to explain what we were aiming for. During our editorial review of initial drafts, we realized researchers were struggling to fulfil our request and appreciated the groundbreaking efforts of Handbook contributors. Together, Sections 3 and 5 contribute essential descriptions of cultural adaptations of mixed methods research and illustrate new ways of designing culturally appropriate mixed methods research. In offering practical guidance transferable to study contexts beyond those described in the Handbook, I call for researchers to advance practice, in these emerging areas, through context- or cultural-specific methodological and theoretical discussions. I highlight the innovative efforts of Jamelia Harris (2022) to offer “lessons from the field” gleaned from mixed methods research experiences in six countries across Africa and the Caribbean as a guiding example providing a methodological discussion of a specific context. It is my sincere hope that this Handbook enhances the visibility of diverse global cultural contexts in which mixed methods research takes place (see also Section 5 Conclusion). I suggest, as a global community of mixed methods researchers, that we commit to selecting publications that feature less familiar research contexts and cultures for reading and citing in our work. Let us diversify the contexts we feature in our selections of mixed methods-focused books and articles to read and also to suggest to others as assigned course readings and citations. Let us promote making explicit our own paradigmatic assumptions (Mertens, 2023) and the contextual influences in our design descriptions, and encourage the practice as we review the work of others. Let us glean insights from our perspectives of how theories, cultures, participants, settings and researchers influence the design of mixed methods research. As we reflect and share with others, we can increase our own awareness of the dynamic contextual influences on mixed methods research design.

HANDBOOK CONCLUSIONS

REPRESENTING DIVERSE MIXED METHODS RESEARCH DESIGN PERSPECTIVES The perspectives that researchers see in published mixed methods research design can have important consequences on future research teams and participants. For researchers wanting to form research collaborations, they might seek guiding examples of various team configurations (see Chapters 16 and 35). For researchers seeking particular participants, they might seek guiding examples of sampling, recruitment, and protocol procedures (see Chapters 9 and 25). Among the key challenges encountered by researchers is access to information and guidance in mixed methods research design descriptions in order to generate publications that are more inclusive of diverse perspectives. Researchers can be discouraged from undertaking mixed methods research designs if they do not see themselves, or the participants they seek to include, represented in the literature. I see four key actions as potential disruptors to the lack-of-diverse-representationin-publications-cycle through focused advocacy for showcasing opportunities, recruitment of peer reviews with lived experiences, accounts of procedures and outcomes, and efforts to enhance readership. One way to disrupt the cycle is to advocate for showcasing diverse researcher and participant perspectives in mixed methods research designs through earmarked opportunities at conferences, in publications, and with funding agencies. In Chapter 1, I describe my editorial efforts to include diverse perspectives in this Handbook in terms of authors’ career stage, geographical location, research context and areas of expertise. I am grateful for the input afforded by Section Leads, International Advisory Board members and global mixed methods research members through the community-sourcing approach to bring the diverse group of 78 authors and their design topics to the Handbook. I suggest the use of a broad community-sourcing approach in future publications. To increase the number of refereed publications and conference sessions, I draw attention to the need for favourable reviews from peer reviewers who recognize and value diverse perspectives from both researchers and participants. To that end, I advocate for the recruitment of peer reviewers, possibly with lived experience, to evaluate publications, conferences and funding proposals that are inclusive of diverse perspectives. Such expertise would enhance the likelihood of equitable treatment. Supporting journals

545

that publish diverse perspectives by giving them our time as peer reviewers is also a worthwhile investment in creating a more inclusive community. My extensive reviewer experiences have greatly expanded my familiarity with mixed methods research designs that are inclusive of diverse perspectives and have informed my own development of more inclusive designs of mixed methods research. If you are new to the field, I recommend you contact the editor for a journal you wish to contribute to and let them know of your interest to review. Effectively conveyed, detailed descriptions of researchers’ backgrounds, as well as those of participants and their selection, recruitment and involvement can offer essential access to the unique perspectives that researchers bring to their research individually and collectively as a team. I offer evidence of my efforts in Chapter 36 to make explicit the individual and collective perspectives of our three-member collaboration involving myself, John W. Creswell and Peter Rawlins. In describing our different backgrounds and roles during the development of this Handbook, we sought to make explicit the diversity in our individual perspectives and contributions to our collective outcomes. My numerous research team experiences have provided insight into how effective teams are formed and sustained (Poth, 2019) and I encourage descriptions for others to learn from. I recognize that word count limitations and prescribed structures in proposals and publications can constrain researchers in providing this information. I advocate for expansions in word counts in our mixed methods research publications and consideration of expanding the formats and audiences for publications we cite beyond what is typical (i.e., peer-reviewed publications) to support the movement towards public scholarship and expand the readership of mixed methods research. Finally, to help researchers locate mixed methods research publications inclusive of diverse perspectives, I advocate for the intentional use of specific labels or phrasing to identify the perspectives in the titles, abstracts and keywords. Identifying all the perspectives may not be possible or desirable in all the suggested locations. Instead, researchers should include identifiers in any of the locations based on what is possible in their publishing outlets. I draw attention to the use of phrases in Handbook chapter titles to identify participant perspectives: for example, “Integrating … from Ethnically and Racially Minoritized Groups…” (Chapter 25). I suggest that identifiers of design perspectives can help make future publications findable for researchers.

546

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

ASSIMILATING PRACTICE EVOLUTIONS IN MIXED METHODS RESEARCH DESIGN EDUCATION Conceptualizing the necessary design expertise as evolving and changing has implications for what we focus on learning as mixed methods researchers and how we go about teaching as mixed methods research instructors. As I attempt to address the query often asked to me, “What do I need to know and be able to do to design mixed methods research?” I openly acknowledge that it represents an enduring question. Consensus has not yet been reached among the global community of mixed methods researchers and because of the changeable nature of mixed methods research expertise, reaching consensus may not be possible or desirable. I also wish to acknowledge the growing scholarship of mixed methods research-specific teaching and learning literature over the past two decades (e.g., Creswell et al., 2003; Greene, 2010; Guetterman, 2017) that has influenced and continues to influence my thinking. Among the key challenges encountered by those teaching or planning to teach mixed methods research is the lack of practical guidance informing instruction that assimilates practice evaluations in real-time, priority afforded to preparing researchers to be capable of future design innovations, and attention given to supporting learner-centred development of design expertise. In speculating about the future of mixed methods research education, I advance the need for keeping pace with design practice evolutions in mixed methods research, anticipating areas of future-forward mixed methods research design expertise, and sustaining individualized progression towards design proficiency specific to mixed methods research. Keeping pace with design practice evolutions emanating from the field is vital to maintaining the relevance of mixed methods research education. I have seen firsthand over the last decade some of the evolutions in mixed methods research practices that have emerged in response to researcher needs in the field that have now become part of the expertise required for mixed methods research design. For instance, the use of visuals in the design of mixed methods research might seem to be an established practice—especially to newcomers to the field—yet it represents a relatively new practice that has only recently become common practice. A key contribution of design diagrams is making explicit the requisite integration of qualitative and quantitative perspectives in mixed methods research. The inclusion of design diagrams in recent publication guidance might lend the illusion of a long-ago established practice; for

example, in Creswell & Plano Clark’s (2018) writing structures and Fetters and Molina-Azorin’s (2019) checklists of mixed methods elements. Design diagrams provide an important example of a recent mixed methods research practice evolution that would now be accommodated within the scope of expected design expertise for a researcher to know and be able to do. Indeed, developing the knowledge and skills required for visuals has become a staple topic in my own teaching of mixed methods research. In this Handbook, we see examples of the use of visuals in Chapter 8 where Schoonenboom advances a strategy for visualizing the interactions between methods and inferences. Also embedded throughout Section 4 (Chapters 22–26) are discussions of software applications for visualizations supporting innovative integrations (see also Section 4 Conclusion). A broad audience can benefit from ShannonBaker’s (Chapter 32) innovative use of visuals to teach and learn mixed methods research. When I first started teaching a doctoral mixed methods research course more than a decade ago, I could not have predicted how my approach and the skills I teach would evolve to reflect practice advancements emerging from the field. I speculate that the pace of evolutions will increase because the fallacy of mixed methods research design expertise as a fixed definition was motivated by an imagined-to-be-ideal rather than being a definition rooted in reality. Education needs to keep pace with what researchers need to know and be able to do in a rapidly changing world. Anticipating areas of future-forward mixed methods research design expertise is essential for preparing researchers capable of design innovations. As an early advocate of adaptive practices, I have seen first-hand the need to prepare mixed methods researchers to think creatively about the practice dilemmas they encounter. At the time I was interested in designing mixed methods research for what others were calling “wicked” problems—defined as those “that involve multiple interacting systems, are replete with social and institutional uncertainties, and for which only imperfect knowledge about their nature and solutions exist” (Rittel & Webber, 1973, as cited in Mertens, 2015, p.3). With no known solutions, I realized the need for rethinking some of the established design practices when faced with greater complexity. In my book, Innovations in mixed methods research: A practical guide to integrative thinking with complexity (Poth, 2018), I described six adaptive practices for responding appropriately to varying conditions of complexity in our mixed methods research. I did not know it at the time, but I was advancing future-forward design guidance evidenced by the increasing recognition of sources

HANDBOOK CONCLUSIONS

of complexity and the usefulness of complexity theories for mixed methods research (Kallemeyen et  al., 2020). Several Handbook chapters offer future-forward orientations to guide researchers in the situational know-how for mixed methods research design addressing complex (or wicked) problems (Chapters 4, 9 and 20). Looking ahead, I anticipate additional emerging practices such as the innovative use of technology applications (see Section 4 chapters and section Conclusion), digital technologies (see Section 6 Conclusion), joint displays as an integration technique (Fetters & Guetterman, 2021), and community-involved approaches (Chapter 15, 27 and 29) as likely expertise that will be included in future delineations of what mixed methods research designers need to know and be able to do. Sustaining individualized progression towards design proficiency specific to mixed methods research specific to mixed methods research is necessary for supporting learner-centered educational approaches. Experience tells me that the diverse backgrounds and experiences of mixed methods research learners requires rethinking of our one-size-fits-all training approaches. Learners are not tabula rasa, also described as “blank canvases”, and it should be expected that mixed methods research learners draw upon their diverse research backgrounds and disciplinary experience and expertise. I advocate for a customized approach where the mixed methods research learning is either entirely or partly tailored to respond to individual learners’ needs, priorities and goals. A learner-centred approach positions the learner to make connections between the experience and knowledge they already have and new information they learn specific to mixed methods research, and then apply it in practice. When learners gain practical experiences, they become more proficient as mixed methods researchers (Guetterman, 2017) and more able to contribute to design theory and practice innovations. To fulfil their design potential, learners need early and continuous exposure to mixed methods research in formal undergraduate and graduate programme coursework. In introductory research design courses, mixed methods research should be introduced alongside qualitative and quantitative research. Such courses that do not should be updated. This is necessary to normalize mixed methods research as a legitimate and established research approach with its own (and exponentially growing) body of literature. It is also imperative to create specialized mixed methods research courses in the same way I see advanced quantitative and qualitative coursework. This helps to distinguish the specialized expertise that is necessary to undertake mixed methods research and that

547

helps the field to move beyond the assumption that having expertise in qualitative and quantitative research is sufficient. Of course, I recognize that these types of institutional changes are difficult. I know this well as more than a decade ago I launched the inaugural mixed methods doctoral course at my own institution, and even though this course fills to capacity at each offering and the demand for an advanced course is high, I have not yet managed to launch the advanced course— yet. But the time I spend mentoring students formally in advanced independent study courses, as research assistants and in member supervisory or examining committees as well as informally, to the extent possible, is well spent for the good of preparing them for their own mentoring roles. I am proud that a member of the first doctoral mixed methods course I taught in 2011 is a contributor to this Handbook. I leave it to you to figure out who that was, but they were exceptional as a course learner more than a decade ago and it is a true delight to watch as they now make important contributions as a faculty member in a peer research institution. This is an example of how our global community members will bring about massive shifts in mixed methods research design education over time.

CALL FOR CREATIVE AND OPEN MIXED METHODS RESEARCH DESIGN AND EDUCATION CONVERSATIONS As this Handbook journey ends, it is my hope that the ideas advanced in this Handbook endure in the work that is taken up by others. While the topics in and contributors to this Handbook naturally evolved over time from what was initially proposed, I am confident that the ideas presented will stimulate rich conversations and mixed methods research design innovations. I advocate that both creativity and openness are vital for inspiring the design of mixed methods research applicable for global contexts and the education of mixed methods researchers capable of design innovations for the yet unknown future. I admit that the way forward is not yet clear, but I am confident that inspiring design innovations necessitates attending to the evolving landscapes of design terminology, many influences on design practices, diverse representations of design perspectives and practice evolutions in design education. I call for the global community of mixed methods researchers to promote the creativity and openness necessary for realizing the mixed methods research design innovation potential discussed in this Handbook. Are

548

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

we willing and prepared for such conversations in our mixed methods research design and education initiatives? I look forward to what the future brings!

REFERENCES Bazeley, P. (2018). “Mixed methods in my bones”: Transcending the qualitative-quantitative divide, International Journal of Multiple Research Approaches, 10(1), 334–341. https://doi.org/ 10.29034/ijmra.v10n1a22 Bergman, M. M. (2008). The straw men of the qualitative-quantitative divide and their influence on mixed methods research. In M. M. Bergman (Ed.), Advances in mixed methods research (pp. 11–21). Sage. Creswell, J. W., Tashakkori, A., Jensen, K. D., & Shapley, K. L. (2003). Teaching mixed methods research: Practices, dilemmas, and challenges. In Tashakkori A., Teddlie C. (Eds.), The Sage handbook of mixed methods in social and behavioral research (pp. 619–638). Sage. Fetters, M., Wu, J. P., & Chandanabhumma, P. P. (2021). Words matter: Calling on the community of research to recognize, react to, and remove racializing research rhetoric. Journal of Mixed Methods Research, 15(1), 6–7. https://doi.org/ 10.1177/1558689820977233 Fetters, M. (2022). A comprehensive taxonomy of research designs, a scaffolded design figure for depicting essential dimensions, and recommendations for achieving design naming conventions in the field of mixed methods research. Journal of Mixed Methods Research, 16(4), 394–411. https:// doi.org/10.1177/15586898221131238 Greene, J. (2010). Foreword: Beginning the conversation, International Journal of Multiple Research Approaches, 4(1), 2–5. https://doi.org/10.5172/ mra.2010.4.1.002 Guetterman, T. C. (2017). What distinguishes a novice from an expert mixed methods researcher? Quality & Quantity, 51(1), 377–398. https://doi. org/10.1007/s11135-016-0310-9 Harris, J. (2022). Mixed methods research in developing country contexts: Lessons from field research in six countries across Africa and the Caribbean. Journal of Mixed Methods Research, 16(2), 165–182. https://doi.org/10.1177/15586898211032825 Kallemeyen, L. M., Hall, J. N., & Gates, E. (2020). Exploring the relevance of complexity theory for mixed methods research. Journal of Mixed Methods Research, 14(3), 288–304. https://doi.org/ 10.1177/1558689819872423

Mason, J. (2006). Mixing methods in a qualitatively driven way. Qualitative Research, 6(1), 9–25 https://doi.org/10.1177/1468794106058866 Mertens, D. M. (2015). Mixed methods and wicked problems. Journal of Mixed Methods Research, 9, 1-6. https://doi.org/10.1177/1558689814562944 Mertens, D. M. (2023). Mixed methods research. Bloomsbury Academic. Mertens, D. M., Bazeley, P., Bowleg, L., Fielding, N. G., Maxwell, J. A., Molina-Azor, J. F., & Niglas, K. (2016). Expanding thinking through a kaleidoscopic look into the future: Implications of the Mixed Methods International Research Association’s task force report on the future of mixed methods research. Journal of Mixed Methods Research, 10(3), 221–227. https:// doi.org/10.1177/1558689816649719 Onwuegbuzie, A., & Poth, C. (2016). Editors’ afterword: Toward evidence-based guidelines for reviewing mixed methods research manuscripts submitted to journals. International Journal of Qualitative Methods, 15, 1–13. https://doi.org/ 10.1177/1609406916628986. Pearce, L. D. (2015). Thinking outside the Q boxes: Further motivating a mixed research perspective. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research inquiry (pp. 42–56). Oxford University Press. Poth, C. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage. Poth, C. (2019). Realizing the integrative capacity of educational mixed methods research teams: Using a complexity-sensitive strategy to boost innovation. International Journal of Research and Method in Education, 42(3), 252–266. https://doi.org/10.1 080/1743727X.2019.1590813 Poth, C. (2020). Confronting complex problems with adaptive mixed methods research practices. Caribbean Journal of Mixed Methods research, 1(1), 29–46. Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155-169. https://doi.org/10.1007/BF01405730 Sanscartier, M. D. (2020). The craft attitude: Navigating mess in mixed methods research. Journal of Mixed Methods Research, 14(1), 47–62. https:// doi.org/10.1177/1558689818816248 Song, M. K., Sandelowski, M., & Happ, M. B. (2010). Current practices and emerging trends in conducting mixed methods intervention studies in the health sciences. In A. Tashakkori & C. Teddlie (Eds.), Sage handbook of mixed methods in social and behavioral research (2nd ed., pp. 725–748). Sage. Tashakkori A., & Teddlie, C. (2003; Eds.), The Sage handbook of mixed methods in social and behavioral research. Sage.

Index action research, 27, 29, 60, 63, 67, 70, 171–173, 247, 423–424, 470, 473, 489–490, 497–504, 507 Adair, J., 204 adaptation, 60, 106, 263, 434, 443, 447, 453, 470, 492, 496–507, 534 adolescents, 8, 120, 121–126, 260–261, 263, 364–365, 453 African-Caribbean women, 195–196, 198–199 agency, 79, 82–84, 87–88, 109, 116, 192, 237, 402, 416–417, 515 Agni, A. 10, 422–430 Aguinis, H., 486 Alavi, M., 518 Alkin, M. C., 399 alternative frameworks for mixed methods designs 69–71 Alvarado, C., 453 American Evaluation Association (AEA), 397 Anderson, J. L., 10, 496–507 Andreotta, M., 157, 160 Anfara, V. A., 284 Anguera, M. T., 10, 394, 395, 446–458, 463 anti-racism, 193 Anttila, M. R., 298 Aotearoa New Zealand, 8 founding document, 220–221 founding of, 220 Te Tiriti o Waitangi, 220–221 Archibald, M., 68, 512–523 Arias-Pujol, E., 446–458 artificial intelligence (AI), 538, 540 Austria, 348 Babchuk, W. A., 291–301 Bakeman, R., 450 Bankier, J., 413 Bansal, P., 489 Bartel, M., 233–241 Bavelier, D., 357 Bazeley, P., 5, 69, 131, 146, 342, 481 Becker, H. S., 40 Bedwell, W. L., 458 Bell, E., 154 Bergman, M. M., 491 Bergmann, M., 512, 513, 519 bias, minimizing, 365–366 Biddle, C., 82 big data, 7, 98–99, 154–161, 181–183, 313, 342, 362, 471, 531–533, 535, 538–539; See also individual entries

Bishop, R., 219, 223, 226, 229 Black feminist epistemology, 195–196 Blau, P., 40 Bonoma, T. V., 270 Boot, C. R. L., 108 Bornakke, T., 157 Borra, E., 157 Botcheva, L., 401 Botha, L., 238 Boud, D., 403 Bowers, B., 239 Bowleg, L., 193 Boyce, A. S., 397–406 braided river, 8, 218–219, 229–230, 411 Breimaier, H. E., 503 Brevik, L. M., 9, 187, 312, 313, 346–358, 388 Brousselle, A., 56 Bryman, A., 30, 523 Bullard, R., 49 Bulut, O., 160 Burke, R. B. 285 Bustamante, C., 372–384 Busetto, L., 106 Butler, P., 218–230 Cahill, D., 413 Cahill, J., 79–89 Cain, L., 154, 155 Cameron, R., 154–161 Campbell, D. T., 22 Campbell, L., 348 Canada, 234, 503, 529 Canales, G., 236 Caracelli, V. J., 22, 23, 68, 151, 248 Caribbean studies, 187, 198 Caruth, G., 68 case-oriented merged analysis, 9, 372–373, 375, 377–385, 533 case-oriented visual displays for, 380, 384 selection of exceptional cases, 378 case study-convergent mixed methods, 373 professional development, Web 2.0 Spanish teachers, 373–377 case study-mixed methods design, 385 case study mixed methods research (CS-MMR) application, 273 design, 273–274 intersection, 270–272 type of case study, 273

550

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

case study research (CSR), 267–269 boundaries of the case, 269 case selection, 268–269 designs, 269 history of, 268 Cauwelier, P., 240 Central Alberta, Canada, 498 Central America, 427 Chandanabhumma, P. P., 422–430 characteristics, mixed methods study, 22–23 Charmaz, K., 329, 331, 335 Chilisa, B., 235 China, 10, 434, 435, 437. See also individual entries cultural context of research, 435–437 East China, 69 illustrative project, 437–441 China Rural Health Initiative (CRHI), 434 Chiu, L.-F., 196 Christ, T., 523 Christensen, L. B., 134 Christie, C. A., 106, 399 Church, A. T., 204 cigarette smoking, 8, 188, 191–200 illustrative study, 194–199 Clark, D., 357 Cole, E. R., 193, 197 collaborative research, 236 interdisciplinary, 236–237 multidisciplinary, 237–238 transdisciplinary, 238–239 Collins, K. M. T., 135, 137, 138, 139 community-based participatory research (CBPR), 10, 54–55, 225, 234, 263, 422–425, 427, 429, 489, 497–498 community-based research, 173, 219, 225, 235, 429 community-benefiting educational programme, 225–227 complementarity, 8, 31, 38, 102, 144, 271, 403 complex designs, 9, 18, 26–27, 31–33, 67, 91, 93, 313, 316, 463, 531 complex interventions, 8, 245–247, 249, 251, 435, 437, 438, 440–441, 443–444, 498, 504, 533 complexity, 5, 32, 70–71, 97, 146, 151, 169–170, 191–193, 199, 267–268, 278–279, 316–317, 349, 351, 485, 487–488, 491–492, 517, 521, 546–547 complex research designs, 126, 154 complex research issues, 267, 270 computer-assisted qualitative data analysis software (CAQDAS), 329, 333, 336 conceptual map of discourse development, 7, 18–19, 79–81, 83–85, 88, 92 concurrent nested design, 23 concurrent transformative design, 23 concurrent triangulation design, 23 consolidation, 99, 143, 145, 147–150, 182, 342, 519 constructivist grounded theory, 175, 312, 328–330, 332–333, 335–336

convergent designs, 9, 26, 295, 372–385 Cook, L. D., 267–276 Cook, T. D., 43 Corbin, J., 330 core designs, 18, 26–27, 29, 33, 67, 306 Corrigan, J. A., 130–141 Corvin, J., 426 Cox, K., 321 craft, 7 attitude, 97–98 design decisions, 180–181 mixed methods design, implications, 182–183 crafting research designs, 98–99 Cram, F., 241 Creamer, E. G., 32, 69, 166–177, 293, 294, 300, 301, 393–396 Crenshaw, K., 192 Creswell, J., 21–31, 198, 527–536 Creswell, J. D., 42 Creswell, J. W., 17, 29, 31, 37, 42, 63, 65, 66, 68, 69, 72, 86, 106, 237, 239, 259, 270, 284, 305, 316, 412, 450, 464, 516 critical dialectical pluralism, 99, 131–132, 138 critical race theory, 247, 361–364, 367–368, 463 Cuba, 503 cultural diversity, 10, 205, 305, 399–400, 429, 434–437, 441, 444, 534 culturally relevant mixed methods design typologies, 306 culturally relevant teaching approach, 398 culturally responsive design, 401, 405 cultural narrative framework, 10, 12, 411–413, 415–417, 534 Japanese cultural narrative framework, 413 cultural reflexivity, 463–464 cultural responsiveness, 233–241 enquiry, 401 mixed methods enquiry, 404–406 mixed methods evaluation design, 401–402 pedagogy, 398 relationships, 51–52 thinking and practice, 398–400 training, 465 cultural values, 203–204 instrument, 215 limitations, 204–205 cultures, 9–10, 19, 72, 200, 203–206, 214–215, 221–222, 233–234, 236, 240–241, 305–307, 393–402, 404–406, 411–412, 425–426, 434–438, 440–441, 444, 446–447, 450, 452–453, 457–458, 463–465, 513–515, 531, 544 cultural competency, 400, 465 cultural diversity, intervention design, 434–444 cultural adaptation, 443 culturally sensitive implementation, 444 methodological dialogues, promoting, 464 methodological writing, challenges, 396

INDEX

mixed methods thinking and practice, 464–465 research design, 394 research design, in mixed method research, 394–395 Curry, L. A., 517, 518, 520 Daigneault, P., 279, 284 Darwin, C., 38 data, 145–147, 150 analysis, 23–24, 27–28, 67, 103–105, 107, 114, 116–117, 119, 121, 122–123, 124, 143–144, 149–150, 180–182, 273, 281–283, 317–318, 324–325, 328–329, 331–335, 339–342, 353, 441 consolidation, 147, 148 integration, 7, 70, 98–99, 143–151, 181–183, 278–279, 342, 373, 375, 376, 384, 533 linkage, 146–147 transformation, 145–146 triangulation, 197–199, 306 types, 144–145, 149–150, 292, 328, 342, 347–349, 351–353, 389 Davidman, L., 196 Davis, S., 256–265 De Allegri, M., 101–111 Dearry, A., 234 DeCuir-Gunby, J. T., 187–190, 305–307, 361, 362, 363 definitions of mixed methods research, 24–25, 517 DeJonckheere, M., 422–430 DeJonckheere, R., 425, 489 Del Giacco, L., 446–458 Delphi, 8, 106, 188, 203, 205–208, 210, 214–216, 435 Denis, A., 193 Denny, E., 196 Denscombe, M., 82, 86 Denzin, N. K., 37, 176 design, 1–13, 21–34, 37–44, 59–72, 97–111, 114–127, 136–140, 166–172, 180–183, 191–200, 246–251, 272–276, 278–287, 291–295, 393–406, 471–482, 496–507, 513–523, 527–536, 542–547 design approaches, expanding, 7–9 design challenges, 4, 10 design concept, 60–63 design conversations, 91–93 design decisions heuristics and tools, 182 design implications, 349 design integration development, mid-20th century, 40–41 early history, 38–39 design-in-use, 39 design issues, 24 design names 23, 24, 26, 31 design possibilities, 10 design trends, 10, 527–529, 531, 533, 535 design typologies, 6, 8, 19, 42–44, 64–66, 68–69, 71, 91, 183, 305–307

551

design visualizations, 114 design flow, visualizing, 115–119 fit of design components, 115–116 specific design components, 116–117 strengths and weaknesses, 119 dialectical paradigm, 166, 169, 172 dialectical pluralism, 99, 104, 109, 131–132, 138, 169, 350, 411 dialogue, 54, 79, 100, 104, 106, 109, 342, 398, 412–414, 416–418, 513, 516, 518 dialogue relational ethic 79 dichotomy, 4, 143, 149–150, 329 Dickinson, W. B., 321 disabilities, 131, 154, 158, 400, 478, 490 discourse development, conceptual map, 83–84 application, 85–87 overview, map, 84–85 The Discovery of Grounded Theory, 292 dissemination, 234, 239, 422, 424, 426, 464, 470, 487, 491, 496–502, 505 diverse mixed methods research design perspectives, 545 diversity, 1, 6, 10–11, 17–18, 79, 91–93, 194, 256–258, 263–264, 398–401, 434–437, 441, 444, 464–465, 489 diversity, equity and inclusion (DEI), 6–7, 18, 11, 13 Dobinger, J., 55 Document Map, 319 Douglas, J., 191–200 Doyle, L., 68 Dubois, E., 157 DuBois, W. E. B., 39 Duchon, D., 170, 171, 174, 270 Due, B. L., 157 Eckert, P., 41 economic justice, 49, 55–56 Edmondson, A. C., 240 education conversations, 547–548 Edwards, C. D., 166–177, 293, 294 emergent designs, 63, 98–99, 101–106, 108–111, 181, 183 emergent mixed methods designs, 101–103, 110–111 added value, emergence, 108–109 conceptualization stage, 105–106 data analysis and interpretation stage, 107–108 dialectic pluralism, 104 feasibility considerations, challenges, 110 operational definition of, 104–105 resource availability, challenges, 109–110 sampling and data-collection stage, 107 theoretical foundation of, 103–104 validity and credibility, challenges, 109 environmental justice, 6–7, 18–19, 48–51, 53, 55–56, 488, 533, 539 equitableness, 131, 132 equity, 8, 10–11, 88, 92, 158, 161, 263–264, 396–398, 400–403, 422–426, 428–430, 490, 492

552

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

equity-focused, 10, 396, 398, 400–401 Escalante-Barrios, E. L., 3, 463–465 ethicality, 131, 132 ethical reflection, 356–357 ethics, 24, 49, 91, 98, 154, 155, 161, 180, 181, 538, 539; See also individual entries big data, examples, 158–159 big data, framework, 155–157 big data and implications, 157–158 examples, 160–161 mixed methods design, 181 ethnically and racially minoritized (ERM) populations, 361–369 aggregation of, 364 integration challenges, funding research, 366 integration challenges, research methods bias, 365 integration challenges, small sample sizes, 364 mixed methods research, implications, 367–368 sampling solutions, 364–365 ethical research, 154–155 ethnic, 54, 154, 191, 193, 195–197, 204–206, 208, 210, 228, 362, 364–366 ethnography, 22, 60, 63, 70, 136, 182, 275, 282, 301, 306, 328 Europe, 39, 54, 349 evidence-based practice, 81, 297, 470, 497–498 exploratory sequential, 27, 31, 67, 205, 267, 272–276, 293, 295, 305, 316, 366 exploratory sequential case study design, 274 qual and quan data, 274 The Expression of the Emotions in Man and Animals, 38 Fàbregues, S., 3, 17–19, 91–93 Fetters, M. D., 22, 30, 31, 82, 168, 177, 270, 272, 323, 342, 350, 389, 411, 485–493, 529, 543 Fisher, P., 80 Flecha, R., 54 Florida, 53 focus groups participants, 137, 147, 196, 198 Ford, H., 157 Frels, R. K., 131 Freshwater, D., 19, 79–89, 342 fully integrated mixed method research, 176 future projections, 188 Gall, M. D., 269 game-based research, 9, 272, 313, 346–347, 349, 351, 353, 355–357, 387, 490, 534 game-based research integrations, 346–358 completion and explanation, 348 corroboration, 347–348 development and infusion, 348–349 implications, 357–358 innovation, 349 prevalence of and potential for, 347 Garnett, B. R., 297, 301 Gay, G., 398 Germany, 447

Gerring, J., 267 Gibbons, S. W., 298 Glaser, B. G., 292, 294, 329 Glewwe, P., 377 global mixed methods research community, 4 global pandemic, Covid, 19, 7, 33, 109, 134–137, 139, 160, 189, 227, 234, 274, 275, 424, 428, 471, 477 Gobo, G., 492 Gómez, A., 236 Gone, J. P., 393 Goodyear-Smith, F., 411 Grace, D., 193 grand challenges, 10, 60, 71, 469–471, 485–486, 488–493, 533, 535, 539 of COVID-19 pandemic, 491–492 equity promotion, marginalized populations, 490–491 of migration, 491 poverty, unemployment and environmental degradation, 490 Grandy, G., 273 Granikov, V., 102, 108 Grant, V., 426 Greene, J. C., 22, 23, 24, 42, 68, 70, 92, 102, 151, 168, 234, 235, 248, 346, 347, 349, 350, 353, 357, 403, 492, 516, 520 Griffin, A. J., 155 grounded text mining approach (GTxA), 9, 328–343 constructivist origins, 329–330 epistemological stance, 332 four procedures of, 333–334 implications, 342–343 iterative process, 331–332 key ethical issues, 341–342 key terms and descriptions, case study, 332 meta-inferences, 340–341 mixed methods origins of, 330–331 nursing researchers, case study, 332–333 overview, 331 qualitative data analysis, 335–336 quantitative data analysis, 336–339 textual data, overview, 334–335 grounded theory, 7–9, 98–99, 166–177, 180, 182–183, 187, 189, 282–284, 291–295, 297–301, 312–313, 318, 328–336, 339, 341–342 Guba, E. G., 42, 247, 325 Guest, G., 32, 136, 139 Guetterman, T. C., 4, 176, 270, 272, 291–301, 311–313, 323, 330, 357, 387–389, 492 Hall, B., 32 Hall, J. N., 397–406 Hamamatsu, Japan, 12 Hamel, J., 268 Hankivsky, O., 193 Hanson-Abromeit, D., 299 Hardy, G., 85

INDEX

Harnett, M., 538–541 Harris, J., 109, 452, 544 Hartnett, M., 218–230, 469–471 Harvard, 22 Hatta, T., 411–418 Hawaii, 529 Hay, M. C., 43 Haynes-Brown, T. K., 478 health disparities, 53, 55, 394, 423–424, 427 health equity, 10, 225, 396, 422–426, 428–430, 499, 533 Helman, C., 463 He Awa Whiria, 8 Hemmings, A., 517 Herbey, I. I., 496–507 Herrmann, H., 154–161 Hesse-Biber, S. N., 155, 237, 239, 306, 517 hierarchical linear modelling (HLM), 322 Hillard, D., 156 Hinds, J., 413 Hitchcock, J. H., 29, 138, 233, 322 Ho, D.Y.F., 204 Holloway, I., 82 Holm, T., 357 Hong, Q. N., 102 Hongling Chu, 434–444 Horvath, K., 403, 491 Hossler, D., 22, 24 Howard, K., 32 Howell, M., 291–301 Howell Smith, M. C., 175, 176, 294, 299 Hoy, R. R., 204 Huberman, A. M., 21, 325 Huieming Liu, 434–444 Husserl, E., 258 hybrid mixed methods design, 351–352 Iceland, 447 implementation, 7–8, 23, 29–30, 68–71, 238, 297, 299–300, 321, 325, 394, 404, 427–428, 434–435, 437–438, 440–441, 443–444, 464–465, 488–489, 496–505, 523 implications, mixed methods designs, 42–44 Inaba, M., 328–343 inclusivity, 393 Indigenous, 4, 8, 10, 49, 55, 203–221, 229–230, 234–235, 238, 411, 534 Indigenous cultural values, 203–216, 214–215 Indigenous people, 49, 55, 216 Indonesia, 50, 55 Indonesia, West Java, Bandung, 50 innovation, 4–7, 79, 93, 150–151, 311, 313, 317–318, 323, 325, 346–347, 349–351, 357, 362, 491–492, 498–507 innovative integrations, 9 forms of, 317–318 illustrative example, 349–357 innovative mixed methods designs, 64, 66, 69–71 institution-wide research project, 227–229

553

instrument development, illustrative example, 205 correspondence analyses, 210–214 Delphi study, 205–206 experts and sample results, 210 field-test study, 210 panel of experts, 206 instrument fidelity, 135 integrated methodology coherent approach, multilevel mixing, 168 scaffolding, constructing, 168–169 integrated mixed analysis, 143, 144–145 data analysis, 143–144 integration, 4, 7–9, 23–32, 37–41, 69–72, 97–99, 105–110, 143–151, 166–177, 180–183, 261–263, 278–279, 311–325, 346–357, 363–368, 387–389, 417–418, 469–481, 512–519 codes as categorial variables, 318–320 game-based research, 313 initial separate analysis, 147–148 joint displays, 322–324 meta-inferences and insights, 323–324 mixed methods design process, 181–182 parsing the concept, 30–31 quantitizing and statistics use, 321–322 research question and design, 147 role of software, 316–317 systematic compilation of analyses, 323–324 technology, untapped potential, 387–389 understanding, 30 value of software for, 311–312 integration strategies, 31, 99, 143, 145, 312–313, 315–316, 324, 355, 357, 387–388, 474 access to the key population, 352 data visualization, 355 ethical consideration, 356–357 innovation purpose of, 351 integrated data analysis, 353–355 integrated hybrid design, 351–352 mixed methods way of thinking, 350 stepwise data collection, 352–353 students as co-researchers, 350–351 unlocking metaphor, 355–356 interaction process analysis, 148 interactive design approaches, 6 interpersonal context, 233, 236, 240 interpretive consistency, 140–141 intersectionality, 8, 137, 188, 191–197, 199–200, 306, 362, 534 intersectionality-informed mixed methods research design framework, 199–200 distinguishing features, 193–194 existing approaches and gaps/influences, 192–193 intersectionality research, 192 intervention study, 29, 137, 434 intrinsic case study, 273 Israel, B. A., 424, 426, 428 Italy, 348, 447 Ivankova, N. V., 27, 118, 119, 121, 126, 234, 496–507

554

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

Jacob, S., 279, 284 Jahn, T., 513 Jahoda, M., 39 James, B., 43 Japan, 396, 414, 417, 418, 429, 463, 529 Japanese cultural narrative framework, 413 implications, 417–418 “Jeter vs. Everett,” 43 Johnson, A. W., 41 Johnson, B., 134 Johnson, N. C., 361–368 Johnson, R. B., 31, 63, 72, 82, 135, 169, 174, 176, 258, 291, 292, 293, 294, 299, 300, 301, 316, 330, 476, 515 Johnson, S. L., 97–100, 180–183 joint displays, 30–31, 33, 114, 312–313, 315, 317–318, 322–325, 388–389, 477, 481, 492, 540, 547 Jones, D. E., 425 Jordan, T. R. 233–241 justice, 48 economic justice, 49, 55–56 environmental justice, 49, 55–56 social justice, 48–49, 55–56, 82–83 Kaczmarczyk, P., 491 Kakai, H., 328–343 Kamalodeen, V. J., 267–276 Kaplan, B., 170, 171, 174, 270 Kaupapa M¯aori research principles, 218–230 community-focused research approaches, 224–225 principles of, 223–224 research with, 221–223 Kemp, L., 131 key elements, mixed methods design, 23 Kim, D., 496–507 Kirmayer, L. J., 393 ki-shou-ten-ketsu, 411–419 Knight, D., 196 knowledge production, 80 knowledge translation, 10, 470, 496–507, 534–535, 539 answering system level questions, 505 designing studies, context, 504 insights and perspectives, 503–504 mixed methods research design considerations, 505–506 process and outcomes, 504–505 Kobayashi, H., 413 Komesaroff, P., 414 Krueger, R. B., 297 Kuckartzand, Udo, 315–325 Kuhn, T. S., 42, 81 Kvale, S., 257 lack of consensus, 18 Latcheva, R., 403, 491 Latin American countries, 452 Leech, N. L., 139, 183, 285

legitimation, 4, 8, 64, 222–223, 226–227, 229, 278, 284–285, 287, 295, 299–301 Le Play, Frederic, 268 Lewin, G., 502 Lewin, K., 429 Lewis, G., 192 LGBTQI, 55 Liebenberg, L., 237, 238, 278, 279, 281, 285 Lieber, E., 145, 150 Lievens, B., 158 Lilley, Spencer, 218–230 Lincoln, Y. S., 37, 42 Linguistic Variation as Social Practice, 41 literature review, 23, 68, 106, 175, 194, 352, 479 key elements, 23 Lithuania, 348 Lohmann, Julia, 101–111 Loomis, D., 32, 102, 104 Love, B., 52 Lucero, J., 55 Lunde, Å., 513, 517, 518, 520 Lynam, T., 30 Macfarlane, A., 223 Macfarlane, S., 223 Machleid, F., 299 Malawi, 13 Malaysia, 8, 187, 188, 205, 214 Malaysian cultural context, 205 Malinowski, B., 41 Malinowski, Bronislaw, 268 Mannell, J., 245–251 mapping, 10, 64, 87, 146, 278, 284, 287, 471, 473–474, 521, 527–536 marginalized and vulnerable populations, 49, 50, 53, 55, 56 Marienthal, 39 Martel, R., 411, 413 matching, 31, 138, 140, 183, 295, 297 matrix of codes, 447, 450 MAXQDA, 9, 312, 315, 317–319, 322–325, 329, 333, 336, 492, 522 Maxwell, J. A., 32, 37–44, 102, 104, 114, 115, 117, 119, 121, 122, 126, 182, 393 Mayoh, J., 256–265 McAuliffe, E., 240 McCall, C., 166–177 McCall, L., 192, 196 McCarley, S., 427 McCrudden, M. T., 119–120, 121, 124, 126, 127 McDavid, J., 56 McKinley, C. E., 426 McMahon, S. A., 107 McTigue, E. M., 119–120, 121, 124, 126, 127 Mead, H. M., 223 Mertens, D. M., 19, 48–57, 71, 81, 83, 235, 241, 278, 399, 485, 490, 492, 529, 532, 544

INDEX

methodological paradigm, 80 methods-inference map, 7, 98–99, 114–127, 181–183 design components, 121–122 design components across levels, 122–123 empirical example, 119–120 meta-inference development process, 124–126 overview, 121 research strand, 119, 121–126 Michigan, 22 microanalysis of dialogue, 413 Miles, M. B., 21, 325 Milgram, S., 40 Miller, R. L., 54 minoritized groups, 9, 131, 195, 313, 361, 363, 365–367, 400, 404, 545 Mirza, H., 195 mixed method grounded theory (MM-GT), 7, 98–99, 166–167, 330, 534 embedded approach, constant comparative method, 172–174 integrated methodology, multilevel mixing, 168 methodological integration, 176–177 methodology, 167–168 qualitative and quantitative strands, 171 research designs, 169–171 role of verification, 174 uneasy alliances, 174–175 mixed methods, 1–13, 17–19, 21–34, 37–44, 48–56, 91–93, 97–111, 114–151, 180–230, 233–241, 267–276, 328–336, 361–369, 393–406, 411–418, 469–482, 485–493, 496–507, 512–523, 527–547 historical developments, 59, 474 historical overview, 19, 60, 71, 464 historical perspective, 37–44 mixed methods analysis, 144, 149, 203, 312, 317, 319, 325, 331, 341–342, 388 mixed methods case study research (MM+CSR), 267, 269–270, 272, 276 designs, 272–273 evolution of, 269–270 mixed methods community-based participatory research (MMCBPR), 422–430 applicability and transferability, global contexts, 429 health equity, United States, 424–425 intersection, 425–430 researcher background and approach, 423–424 mixed methods convergent designs, 372–385 mixed methods crosstab, 319, 320 mixed methods (MM) designs design typologies, 64–66 educational texts, 66–68 flexibility and innovation, 69–71 implications, 71–72 prevalence of typologies, 66–68 prevalence studies, 68–69 understanding, key influence, 63–64

555

mixed methods designs, 6–10, 17–19, 21–34, 37–43, 48–55, 59–71, 91–93, 97–99, 101–111, 180–189, 305–306, 397–399, 401–404, 406, 474–478, 491–493, 496–502, 505–507, 527–529, 531–536 mixed methods designs, changing, 24 central elements, designs, 26 complex designs, 27 core and complex design diagrams, 29–30 core characteristics and definitions, 24–25 core designs, identifying, 26–27 design types with diagrams, 27 detailed diagrams, 29 integration, 30–31 naming, designs, 26 visual representations, designs, 27–30 mixed methods design typologies, 8, 183, 305–306 mixed methods evaluation, 8–9, 21, 234, 246–247, 397–398, 401, 403–405, 503 mixed methods-grounded theory (MM-GT), 291–301 characteristics of, 294 conceptualization, 293 contemporary approaches, 293 general characteristics of, 295 grounded theory characteristics of, 295 grounded theory designs, 292 grounded theory in practice, 292 guiding practices, 299–300 key features, published studies, 295–296 limitations, 300–301 matching research questions, 297 methodological references, 296 mixed grounded theory, 294 mixed methods research characteristics, 295 procedures, key characteristics, 294–295 providing rationales for, 297 quantitative data in, 292–293 specifying GT approach, 297–298 synthesis, 294 theory development, embedding dialogic, 293–294 using GT and MMR procedures, 298–299 validity/legitimation and quality indictors, 299 mixed methods instrument development designs, 284–285 mixed methods integration visualization, 312–313 mixed methods phenomenological research (MMPR), 256–265 analytic process, 261 challenges, 262–263 justification for, 258 models, 258 opportunities, 263 phen + quan, 260 PHEN-quan, 258–259 philosophical underpinnings, 260–261 physical activity and quality of life, 260–262 QUAN-phen, 259–260 study methods, 260

556

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

mixed methods research, 1–13, 21–34, 60–88, 97–106, 114–151, 180–183, 187–216, 233–241, 270–276, 305–307, 311–313, 361–364, 366–368, 469–476, 478–482, 485–493, 496–502, 504–507, 512–523, 527–547 mixed methods research design, 1–13, 22–34, 60–72, 86–88, 97–126, 180, 182–183, 191–200, 203–206, 214–216, 219–220, 226–228, 305–307, 387–388, 394, 496–506, 520–522, 527–532, 534–536, 542–547 mixed methods research design conversations, 532 emergent, flexible and uncertain designs, 532–534 implications, 535–536 innovative technology designs applications, 534–535 international applications and cultural adaptations, designs, 534 mapping, 527–528 societal issues with design intersections, 535 mixed methods research design practices contextual influences, 544 mixed methods research design terminology, 543 mixed methods sampling, 130, 132–133, 140, 474 mixed methods teaching, 472–482 mixed methods way of thinking, 349 “mixing” design elements, 403–404 MMIRA task force report, 539, 540, 542 projections, 530–532 Moats, D., 157 Moghaddam, F. M., 234, 236 Molina-Azorin, J. F., 17–19, 22, 30, 82, 91–93, 168, 177, 350, 411, 485–493, 529 Molinero, F., 446–458 Moore, 464 MORE-IC (the Mixed Methods Observational Research for Informed Consent), 413–414, 413–417 mixed methods objective and procedure, 414–416 narrative framework, applying, 416–417 qualitative analysis, 414 quantitative analysis, 414 Morgan, D. L., 22, 23, 86, 173, 176, 514, 515 Morse, J. M., 22, 23, 29, 65 Morse, M. J., 80 Moseholm, E., 30 Müller, J., 157 multi-level mixing, 166, 168, 169, 171, 175 narrative research, 411–418 Nastasi, B. K., 29, 233, 464 New Zealand, 7, 13, 120, 123, 187, 188, 529 Ngulube, P., 69 Niehaus, L., 29 Niglas, K., 19, 59–73 Nisbett, R. E., 418 Norway, 9, 349 “Obedience to authority,” 40 observational methodology, 10, 446–447, 450, 452, 457 O’Cathain, A., 30, 517

O’Donovan, R., 240 O’Fallon, L. R.., 234 ‘Ofanoa, M., 411 O’Halloran, K. L., 156 Olkin, R., 478 Onwuegbuzie, A. J., 82, 130–141, 183, 258, 281, 285, 300, 476, 515 Pakistan, 503 paradigms, 18–19, 24, 37–38, 41–42, 79–88, 175, 181–182, 218–219, 229, 247, 251, 256–258, 263–264, 294, 513–514 paradigmatic assumptions, 134 paradigm wars, 41–42, 85 participant enrichment, 135 participatory methods, 246–247 challenges, cluster RCTs, 250–251 participatory qualitative, 8, 245–247, 251, 498 Parylo, O., 68 Patel, S. G., 476 Patton, M. Q., 22, 269 pedagogy, 373–381, 398, 401, 413, 470, 474, 481, 530 Péladeau, N., 321 Pence, K. G., 298 people of color, 49, 51, 55 personal context, 233–234, 240 phenomenological research, 257–258 phenomenology, 22, 189, 257–259, 262–264, 283, 285, 306 The Philadelphia Negro, 39 physical activity, 8, 116, 260–262, 426 Pieterse, A. L., 330 Plano Clark, V. L., 17, 21–34, 27, 29, 65, 68, 86, 106, 198, 234, 239, 270, 284, 305, 316, 412, 450, 492, 516 Platt, J., 39 Pluye, P., 102, 108 politics, 82–83 Polk, M., 238, 239 Popa, F., 516 Portugal, 348, 447 Poth, C. N., 1–13, 32, 37, 71, 154, 160, 267, 279, 313, 329, 347, 348, 349, 352, 394, 428, 450, 458, 491, 527–536, 542–548 power, 82–83 inequities, 53–55 practice evolutions, 546–547 pragmatic ethics, 132 pragmatism, 24, 26, 29, 42, 82, 101, 103, 110, 134, 513, 515–516 Preissle, J., 154, 155 prevalence studies, 19, 64, 68–69 principles, 8, 61–62, 64, 154–155, 157–158, 160–161, 182, 218–220, 223–230, 362–363, 424–429, 470–471, 486–490, 492–493, 521 process evaluation, 10, 246, 438, 440–441, 443 Prost, A., 245–251 Puffer, K. A., 298 Puigvert, L., 54 Putnam, J.W., 209

INDEX

qualitative data analysis software (QDAS), 322–323, 325; See also computer-assisted qualitative data analysis software (CAQDAS) Q-labels, 149–150 qualitative, 21–33, 37–44, 101–110, 114–115, 118, 119, 124, 126, 134–136, 143–150, 156–160, 169–171, 245–247, 257–264, 268–276, 278–287, 297–300, 315–325, 347–350, 352–355, 414–418, 498–506 qualitative instrument development, 281–284 qualitatively driven mixed methods, 92, 332 quality of life, 8, 48, 256, 260–261, 263 quantitative, 21–33, 37–44, 101–110, 114–115, 117–119, 120, 121, 124, 126, 143–150, 170–174, 195–198, 257–264, 268–273, 278–280, 284–287, 291–294, 297–300, 315–318, 321–325, 347–353, 362–366, 487–492, 498–506 quantitative instrument development, 285 quantitizing, 145–147, 149–150, 285, 287, 315, 318, 321–322, 332, 353–355, 447, 450 race, 188, 191–197, 199–200, 247–248, 306, 361–364, 367–368, 397, 400, 405, 425, 427 racism, 55, 193, 195, 361–363, 366, 368, 399–400, 424–425, 543 Rädiker, S., 315–325 Randomized Controlled Trials (RCTs), 245–252, 533 Rawlins, P., 218–230, 469–471, 527–536, 538–541 realist evaluation, 247–248 realist evaluation research, 247–248 recursive design, 521–523 reflexive practice, 403, 471 Reichardt, C. S., 43 Reinharz, S., 196 relationships, 7–8, 49–52, 56, 116, 146, 155, 191–195, 222–223, 226–229, 238, 248, 261, 298–299, 318, 428–429 representation, 64, 131–132, 137–138, 140–141, 155, 159, 161, 223, 226–229, 388, 481, 516 research as a craft, 97–100 research capacity, 222, 507 research cultures, navigating, 9–10 research design practice, implications, 87 research design, 1–13, 22–34, 37–44, 59–88, 97–110, 114–127, 132–134, 136, 138–140, 166–168, 191–200, 226–230, 305–307, 394–406, 473–476, 496–506, 516–523, 527–532, 534–536, 542–547 research diagrams, 475–477, 480–481, 540 researcher reflexivity, 402–403 research outcomes validity, importance, 278–279 research paradigms, 84, 87–88, 175, 218–219, 229, 247, 270, 276, 514 research practice, 82–83 research strands, 114–127 responsible approaches, grand challenges, 486 practical and social research, 486–488 social research, impact, 486 responsible research, 486–493

557

results-based integration, 316 Rice, C., 199 rigour, 131–132 Rinnert, C., 413 Robinson-Wood, T., 479 Rolfe, G., 83 Roma population, 54 Roussel, L. A., 496–507 Ruiz-Íñiguez, R., 503 Salamonska, J., 491 Saldaña, J., 285 Salem Oregon, 51 Samoa, 8, 187, 189, 246, 248–250 sampling, 7, 30, 32, 97–99, 104–105, 107–109, 130–141, 154–155, 170, 175–176, 180–183, 300, 352, 364–365, 367 sampling design, phases/strands combination, emphasis, and sequence, 138–139 level of mixing, 138–139 other sampling units, 140 relationship, 137–138 sampling technique and scheme, 139 Sanders, K., 492 Sanscartier, M. D., 97, 100, 182, 183, 523, 544 Schafft, K. A., 82 Scherman, V., 278–287 Schilling, N., 41 Schoonenboom, J., 63, 72, 97–100, 114–127, 176, 180–183, 316, 377 Schwartz, S. H., 209 secondary data, 9, 169, 199, 311, 313, 361–368, 389, 491, 531, 533 critical race mixed methodology, ERM groups, 362–363 integrating, challenges and solutions, 363–364 in mixed methods, 362 in mixed methods, ERM groups, 362 Sena Moore, K., 299 sequential explanatory design, 23 sequential integration, 316 sequential transformative design, 23 Shanahan Bazis, P., 294 Shannon-Baker, P., 26, 187–190, 194, 305–307, 472–482, 538 Sharma, G., 489 Sherbino, J., 330 Shim, M., 298, 301 significance enhancement, 135 single methodology instrument design, 279–280 qualitative instruments, development, 281–284 quantitative instruments, development, 280–281 Sinley, R. C., 66, 69, 72, 464 Slate R. N., 52 Smets, A., 158 Smith, L. T., 223 social and behavioural sciences, 473 social contexts, 233–234

558

THE SAGE HANDBOOK OF MIXED METHODS RESEARCH DESIGN

social impact, 10, 71, 278, 486, 515, 523, 539 social injustice, 80, 84, 399 social justice, 19, 55–56, 79, 82–83, 86, 88, 235–236, 341–342, 361–363, 394, 397–402, 404–405, 422–423, 487–488 social justice, 48–49, 55–56, 82–83 software developers, 389 Sorde Marti, T., 490 South Africa, 187, 234 Spain, 13, 54, 446 Spanish research culture, 446–459 challenges, illustrative examples, 457–458 cultural and research context, 446–447 illustrative examples, 452–453 implications, global researchers, 458 integration procedures, 454–456 mediation, conflicts, 457 MMR and design, rationale, 450 observational methodology, 447–450 psychotherapy, 453–454 research gap, 450–452 Stadnick, N. A., 155 Stake, R. E., 268, 269, 399 Stanley, J. C., 22 Steckler, A., 27 science, technology, engineering and mathematics (STEM), 9, 272, 368, 398, 400, 401, 404–406 STEM evaluation, 401, 404–405 Strauss, A. L., 292, 294, 329, 330 The Structure of Scientific Revolutions, 42 students as co-researchers, 350, 357, 490 survey participants, 139, 197 sustainability, planning for, 53–55 Sweden, 238 Swindle, T., 29 Symonette, H., 403 table of specifications, 280 Taghipoorreyneh, M., 203–216 Taiwan, 503 Tashakkori, A., 22, 23, 29, 32, 65, 66, 68, 70, 72, 117, 119, 121, 122, 124, 126, 351 teaching challenges, instructors, 475 data visualization and representation, 481 envisioning future uses of visuals, 479–481 joint displays for integration, 477–479 mixed methods research, 473–475 pedagogy and assessment, 474–475 of research designs, 473 research diagrams, 476–477 use visuals, 475–481 visual data, analysis, 480–481 teaching research methods, 472–473, 475 team interactions, 233, 236, 239–240 teams, 109–110, 189, 233–234, 236–241, 388–389, 428, 430, 469, 471, 474–475, 506–507, 517–523, 535, 539–540, 545

Technological Pedagogical Content Knowledge (TPACK) model, 373, 376 technology, 9 Teddlie, C., 22, 23, 29, 32, 65, 66, 68, 70, 72, 117, 119, 121, 122, 124, 126 Tellis, W. M., 268 text mining, 9, 159, 312–313, 328–336, 341–343, 388, 492, 533; See also individual entries Thompson, T., 256–265, 262 time sequence, 412 Timmis, S., 237 Tobago, 12 transdisciplinary mixed methods research, 10, 469, 471, 512–523, 533 communication and technological considerations, 522 dimensions of integration, design, 518–519 innovative mergers, 521–522 “integration challenge,” 516–517 integrative conceptual work, 519 multidisciplinary, interdisciplinary, and transdisciplinary research, 516 new possibilities, 520–521 paradigms and epistemic cultures, 513–514 paradigms and stances for, 514–515 practical strategies, teams, 519–520 pragmatism and reflexive questioning, 515–516 realist principles and tailored facilitation, 521 recursive design, 522–523 transdisciplinary research, 69, 471, 491, 512–521, 523 transformation, 49–50, 54, 143, 145–150, 182, 189, 316–318, 323, 325, 332, 450, 485, 490 transformation-based integration, 316 transformative assumptions, 49 transformative axiological assumption, 49–50 transformative epistemological assumption, 50 transformative methodological assumption, 50–51 transformative ontological assumption, 50 transformative mixed methods, 48–52, 54, 56, 248, 251, 363, 402, 490 transformative research, 83, 219, 225, 470, 488–490, 493, 539 translational research, 10, 263, 469–470, 496–507, 540 action research, 499 adaptation, 498 additional considerations and developments, 506–507 community-based participatory research (CBPR), 498–499 component synergy, 499 continuum, all stages, 504 dissemination and implementation (D&I), 498 evidence-based practice (EBP), 498 intersecting mixed methods, 499–501 methodological components for, 497–499 MMTR methodological framework, 501–503 translational science, 470, 497, 500–502, 504–507 transparency, 4, 11, 130–131, 138–139, 154–155, 160–161, 300, 315, 317, 492, 522

INDEX

treatment integrity, 135 tree mixed methods sampling, 130–140 TREE (Transparent, Rigorous, Equitable, and Ethical), 7, 99, 130–141 TREE sampling meta-framework, 131–134 description, 134 exploration, 134 objectives for research, 134–135 quantitative vs. qualitative vs. mixed approaches, 135–136 research design(s), 136–137 trunk, 134 understanding, 134 Trinidad, 12 trustworthiness, 131, 278–279, 284–285, 351, 405 Tsheko, G. N., 235 Tsortanidou, X., 297 Turner, L.A., 135 Turner syndrome (TS), 8, 189, 256, 260–262 typology, 8, 18–19, 32–33, 66–68, 70–72, 135, 139, 145, 183, 187, 284, 297–298, 300 typology-based design approaches, 18, 19 typology-based mixed methods designs, 187–190 combination mixed methods designs, 189 cross-cutting discussions, 189–190 cultural considerations, 188–189 underrepresented, 8, 183, 189, 228, 233, 235–236, 238, 240–241, 257, 263–264, 367–368, 399–400, 404–405 Ungar, M., 237, 238 United Kingdom (UK), 52, 63, 187, 191, 195, 198, 246, 463, 464 United States (US/USA), 13, 22, 49, 51, 63, 66, 86, 187, 234, 348, 373, 396, 398–400, 404, 424–427, 429, 446, 463, 464 validity, 2, 4, 32–33, 38, 40–41, 43, 108–109, 144, 203, 214–216, 221–222, 278–281, 285–287, 299–300, 474–475 values, 8, 48–49, 92, 188, 203–216, 222–223, 226, 229–230, 233–235, 316–317, 319–323, 393–394, 396–398, 400–402, 405, 428, 463–464, 492, 498–500, 513–514

559

values-engaged, 10, 396, 398, 400–401 Van Haneghan, J., 321 Van Manen, M., 258 Varjas, 464 Vesper, N., 22, 24 violence against women, 8, 189, 245–246, 248–251 violence against women in Samoa intervention to prevent, 248–249 study design, 249–250 virtue ethics, 132 visualizing research designs, 98, 99, 114–119 visuals, 10, 355, 388–389, 469–470, 472, 475–477, 479–482, 531, 533, 538, 540, 546 Vogl, S., 143–151 Walsh, I., 169, 176, 292, 293, 294, 299, 301 Watkins, D. C., 361–368 Weisner, T. S., 43, 145, 150 Western countries, Global North, 158 Westhues, A., 172, 173, 174, 175 West Midlands, England, 196, 197 wicked social problem, 10, 48, 60, 64, 71, 86, 92, 245, 251, 278, 425, 469, 471, 485, 512, 517, 521, 523, 535, 546 Wickert, C., 486 Widianingsih, I., 50, 490 Wiguna, T., 156 Williams, J., 237 Wilson, A., 399 Wood, M. A., 298 Wray-Bliss, E., 154 Wu, S.-J., 503 Xuejun Yin, 434–444 Yin, R., 267 Younas, A., 282 Zea, M. C., 490 Zentella, A. C., 41 Zhou, Y., 69 Zimbabwe, 54 Zimmerman, L., 278–287 Zoellner, J. M., 427