Doing Academic Research: A Practical Guide to Research Methods and Analysis 0367207915, 9780367207915

Citation preview

Doing Academic Research

Doing Academic Research is a concise, accessible, and tightly organized overview of the research process in the humanities, social sciences, and business. Conducting effective scholarly research can seem like a frustrating, confusing, and unpleasant experience. Early researchers often have inconsistent knowledge and experience, and can become overwhelmed – reducing their ability to produce high quality work. Rather than a book about research, this is a practical guide to doing research. It guides budding researchers along the process of developing an effective workflow, where to go for help, and how to actually complete the project. The book addresses diversity in abilities, interest, discipline, and ways of knowing by focusing not just on the process of conducting any one method in detail, but also on the ways in which someone might choose a research method and conduct it successfully. Finally, it emphasizes accessibility and approachability through real-world examples, key insights, tips, and tricks from active researchers. This book is a highly useful addition to both content area courses and research methods courses, as well as a practical guide for graduate students and independent scholars interested in publishing their research.

Ted Gournelos is Associate Professor at Old Dominion University, USA, specializing in cultural studies, digital media, and strategic communication. As CEO of the consulting firm Story | Strategy, he works with organizations to fuse success and social progress. His broad background in media, culture, and business helps him bridge the gaps between education and application. Joshua R. Hammonds is Assistant Professor at Rollins College, USA, specializing in interpersonal and professional communication. His teaching and consulting work focuses on the management of relationships through both qualitative and quantitative methods. He currently develops research interventions to improve communication competency in the medical industry and increase employee engagement in the corporate world. Maridath A. Wilson is Assistant Professor and E-Resources and Serials Librarian at Rollins College, USA. She specializes in electronic resources management and information literacy instruction to students of the humanities and social sciences. She is interested in issues of constructed authority, as well as the work of demystifying the research process.

Doing Academic Research A Practical Guide to Research Methods and Analysis

Ted Gournelos Joshua R. Hammonds Maridath A. Wilson

First published 2019 by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN and by Routledge 52 Vanderbilt Avenue, New York, NY 10017 Routledge is an imprint of the Taylor & Francis Group, an informa business © 2019 Ted Gournelos, Joshua R. Hammonds, and Maridath A. Wilson The right of Ted Gournelos, Joshua R. Hammonds, and Maridath A. Wilson to be identified as authors of this work has been asserted by them in accordance with sections 77 and 78 of the Copyright, Designs, and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilized in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-in-Publication Data A catalog record for this book has been requested ISBN: 978-0-367-20791-5 (hbk) ISBN: 978-0-367-20793-9 (pbk) ISBN: 978-0-429-26355-2 (ebk) Typeset in Galliard by Apex CoVantage, LLC Visit the eResources: https://www.routledge.com/9780367207939

Contents

List of figuresvii Preface: two professors and a librarian walk into a bar . . . viii

Part I Get the party started right

1

  1

Learn to love, not dread, research

2

  2

Where do I start?

14

  3

Now what do I do? What’s a lit review?

26

  4

Choosing a method

42

Part II Methodology overviews

71

  5

Close textual and thematic analysis

72

  6

Content and discourse analysis

81

vi   Contents

  7

Ethnography, interviews, and focus groups

 8 Surveys   9

Observational methods in practice (participatory-action research, experimental research, theory, and metasynthesis)

Part III Data analysis and the final push

98 119

135

145

10

Analyzing qualitative data and making sense of detail

146

11

Analyzing quantitative data and making sense of numbers

156

12

Persuasion, presentation, and publication

173

Glossary181 Index185

Figures

  1.1 From topic to paper   1.2 Choosing a method   3.1 From topic to literature review (samples from popular culture and society)   4.1 Comparing methodologies   4.2 IRL: How to do what your boss asks   7.1 Types of questions   7.2 Primary and secondary questions   8.1 Populations vs. samples   8.2 Categorical question   8.3 Likert-type scale questions   8.4 Semantic differential question   8.5 Validity and reliability   8.6 Bad questions 11.1 A “normal distribution” 11.2 Negatively skewed distribution curve 11.3 Positively skewed distribution curve 11.4 Tests and variables 11.5 A quick guide to statistics terms

6 9 33 44 67 108 110 121 125 126 127 128 131 159 161 161 165 171

Preface

Two professors and a librarian walk into a bar . . . Research doesn’t usually start in an office, or as an assignment, or while reading some giant textbook. It often starts at the moment you see the need for something, or you take a step back and wonder why something happens the way it does. In other words, research is all about the world we live in and how we understand that world. In our case, this book began, as many books have before it, in a bar. Happy hours are a long tradition in work cultures, and universities are no exception. In this case, Maridath, Josh, and Ted got together after work to have a couple beers and decompress after a long week. While a good happy hour doesn’t dwell on work for very long, it usually starts with a bit of complaining. Ted ordered a pint of some thick stout that looked like an unholy mixture of bread and coffee with some alcohol in it, and told the others how frustrated he was that students do so little research on campus. “They always look at me like I’m crazy when I assign a real research project where they actually gather data. What else would we be doing?” Josh took a sip of his hipster-approved, hop-forward IPA before replying. “Even in research methods classes they aren’t

Preface   ix

prepared for it, and they always end up taking that at the last possible moment because it terrifies them.” “OK, but that’s a problem, right? They’ll be doing this for the rest of their lives, or at least they will if they want to get a decent job. That’s what being a good manager means: research.” Maridath, the librarian that works with Ted and Josh to introduce students to research methods and how to find sources, walked in predictably late and ordered something from Brooklyn to match her artfully distressed leather jacket. “They always come to me for help for exactly that reason, but I don’t know what to tell them. It’s one thing to learn about research. It’s another thing to learn how to actually do research.” They chatted for a while and realized that it wasn’t just their students. It was them, too. Even graduate students have often never learned how to conduct various methods and why you’d choose one over the other for any given question or context. Students, graduate students, and even faculty often just use the same methods over and over again; sometimes they even avoid reading work applying different methods. It was a reality check, as Maridath reminded them. “We can’t blame our students, right? It’s not like there’s a book out there that does this, either. They’re all huge, dense and often boring, and still don’t give you a practical guide.” They all sat for a little while, and then Ted looked up and said, “Um, well, why don’t we just write it?” ***** That all actually happened, more or less. We probably sounded a lot dumber, and probably added in a lot more complaining. But that’s why memory is so great; you can avoid thinking about unpleasant truths. This book is obviously that book, born in a bar and written in offices, coffee shops, libraries, at home, and, again of course, the occasional bar. What we’re going to do here is take you through a few steps. Part 1 (Chapters 1 through 4) is about understanding what

x   Preface research is, how to choose a topic, and how to get the background so that you actually know what you’re talking about and can write a “literature review.” Part 2 (Chapters 5 through 9) takes you through a few of the main methods that you’ll encounter and shows you what they do and why you’d choose them. Part 3 (Chapters 10 through 12) shows you how to actually analyze the data you gathered through your method, and then how to put all of that together into a finished product. While this book is suitable for anyone doing academic research, it’s intended mainly for undergraduate and graduate students. Basically, it’s what we wish we’d read when we were starting out. We’ve added in a couple of things to make this even more useful as a practical guide. We offer some tips and tricks that we use to make things manageable, and we hope those will help you along the process. We also offer “tips from your friendly neighborhood librarian,” in which Maridath takes you through how librarians can help you along each stage of your research project. Chapters also feature “IRL” (in real life) sections that show you how each part of the process might appear in the workplace, in fields ranging from marketing and restaurant management to human resources and customer experience monitoring. Finally, we supply a set of online resources for each chapter. We’ve included articles from many different disciplines that use the methods described in the book, videos that guide you on how to use technology that will make your life a lot easier, and links to citation guides, tutorials, and even paid services that can do everything from transcribing your data to keeping your sources organized to arranging for people to take your survey. We hope you enjoy the book and that it makes the whole process a lot less intimidating. If you’re not interested in your research project, and if it’s not at least a little fun, then you’re doing it wrong. This isn’t something you should dread; it’s a rare opportunity to figure things out, to find out what makes you you, and what makes the world what it is. Run with it.

Get the party started right

PART

I

CHAPTER

1

Learn to love, not dread, research

OVERVIEW AND OBJECTIVES This chapter goes through what research actually is (answering interesting questions), rather than what we usually feel it is (agonizing work). We’ll discuss how to think about your research and your topic, as well as how to find research questions and choose methods. We’ll expand on topics and research questions in the next chapter, but this is the general overview of how to do it without panicking. Using examples of sample topics drawn from a recent class, along with their possible research questions and possible methods, we give you a brief overview of the various kinds of methods you could choose from and that are highlighted in depth in later chapters.

1: Learn to love, not dread, research  3

Starting a research project always sounds a lot harder than it actually is. Most people would rather hide under their desks (or beds, the choice is yours) and hope it goes away than begin a new project. The thought of starting something big, whether a report for your boss, a term paper in a class, or a new book, is often terrifying. We understand that, and we want you to understand why it’s so scary. Research is scary because there’s a whole big wide world out there, and for every piece of that world there are dozens, if not hundreds, of ways to study it. What’s going to work? What will people understand? What won’t drive me crazy to study? What’s interesting about this project, or (scarier) what’s new about this project? We’re getting nervous just thinking about it. Calm down. That’s step one. The good news is that while research is hard, it’s also a lot easier to start than you might think. Because there is so much to study, and in so many different ways, you should be able to find something that interests you, that’s possible to accomplish, and that isn’t completely repeating someone else’s work. That might not seem very reassuring, but it should be. For instance, when Ted began his dissertation for his PhD (not much is scarier than that, we can assure you), he went to a professor with a twelve-chapter book outline that he was really excited about. It studied twelve different media examples (from graffiti to internet memes to The Simpsons). The professor took one look at him, shook his head, and said “you’re insane.” It was true. The project would have taken years and been thousands of pages long if it was done right. So Ted went home, really thought about it, and decided to choose one chapter, the one on South Park, to be the focus for his dissertation and then his first book. This is the question we want you to ask yourself: What can I study that I don’t think will bore me and my readers, or make me hate it? Never start a project because you think it’ll be easy; if you do it right, it won’t be. Start a project because you’re fascinated by it, and then figure out how to make it possible to do in whatever timeframe you have been given.

4  Part I: Get the party started right

IRL: A BOSS THAT WANTS THE WORLD It’s Thursday afternoon and your boss has just come back from a meeting with her boss. She asks to chat with you for a second, which is always scary, and tells you she needs a comprehensive report on how your company is doing around the world. And she needs it tomorrow. At this point you’re more than nervous; you’re worried about keeping your job if you can’t complete an impossible task. So here’s what you do (or what one of us did, since this is a real story): First, ask what your boss really needs to know. In other words, if she just wants numbers that you can find on a company database, that’s not so bad. If she wants those numbers along with detailed information about customer satisfaction, you’ve got a big problem. Second, ask who she needs the research for, and why they want it. If the CEO just wants to be able to show the Board of Directors that you’ve got stores in 20 countries as one tiny part of a presentation, no problem. If they’re evaluating where to invest in new marketing campaigns and where to hire staff to support an international branch, it’s going to be a long, long night and you still won’t finish. Remember that research is a process, and there is always room for more depth or more reach. You have to know what questions you need to answer, and why, before you can do it well. Be careful, because you can get in over your head really fast.

Time is an important factor in research because, in theory, all research projects could go on forever. Every topic is like a huge room that you can’t see into, and each research project just adds

1: Learn to love, not dread, research  5

a little peephole in the wall that you can look through. Maybe eventually you’ll be able to see everything, but it’s going to take you a while, and you’ll have to shift your perspective, depth of focus, and sight angle to see much. And that’s if nothing in the room changes. Good luck with that. People, media, minds, culture, politics – everything is changing all the time. The trick is to get a perspective on what’s happening in a way that meets your goals as well as your timeline. Sometimes that’s a matter of how much you study, and sometimes it’s a matter of how deeply you study it. But time, like money, is never infinite. Once you’ve chosen a general topic, you need to figure out how you want to do the study (see Figure 1.1). Researchers often split our work into three categories or perspectives, or what we call “paradigms”: the Empirical, the Interpretive, and the Critical. Empirical scholars tend to believe that the topic they’re studying is discoverable, and often through measurable data with numbers. They want certainty, and so they tend to focus on large amounts of data that they can extract and analyze in a very methodical way that leaves little room for variation. In other words, they’re looking for a snapshot through that keyhole without drawing conclusions about anything they can’t see. For example, in a project Josh conducted on healthcare, he wanted to know how often patients did what their doctors told them to do. He decided to study whether or not the amount of time doctors spent with patients (the independent variable, or what we use to explain changes) predicted how satisfied the patient was with the experience (the dependent variable, or what actually changes). There are several ways to tackle how time affects patient satisfaction, but for his study, he decided that the length of time and degree of satisfaction told the best story. Interpretive researchers, on the other hand, look at their research slightly differently. They’re not looking for a snapshot, they’re looking for what’s moving. More specifically, they’re looking at how things might be portrayed, received, understood,

Topic 1: Cognitive disabilities in the media

Topic 2: Quality healthcare

Topic 3: Fighting hunger in the USA

Focus: Portrayal of autism and dementia in the media (critical paradigm)

Focus: Doctor/ patient interaction (interpretive paradigm)

Focus: Nonprofit organizations’ use of social media to communicate (critical/empirical paradigm)

RQ1: How do plays portray/ include people with different ways of thinking?

RQ1: How do doctors communicate with patients about health issues?

RQ1: Who do nonprofits tend to target in their social media, and how? How often? With what response from viewers?

RQ2: How do portrayals of autism in children and dementia in the elderly differ?

RQ2: How do patients connect their interaction with doctors to their perception of the quality of care?

RQ2: What stories do nonprofits tell, and what visuals do they use to tell those stories? How often? With what response from viewers?

RQ3: What kinds of problems and solutions do the plays focus on, and who causes/ solves those problems?

RQ3: What might be improved in setting expectations and establishing effective ways of communicating during interactions?

RQ3: How might approaches by various nonprofits differ, and how do they align with industry “best practices” for effective website development?

Figure 1.1  From topic to paper

1: Learn to love, not dread, research  7

Method: Textual analysis of two scenes from two plays (first introduction to a character with cognitive disability, climax of conflict within the play)

Method: Observations of twenty interactions and post-observation interviews with twenty patients

Method: Content analysis of twenty websites, looking at specific categories of visuals, storytelling, structure, and other types of content drawn from industry and scholarly sources on effective websites

Figure 1.1  Continued

and reacted to by the people in that room. For example, you might study how an organization’s staff decides which content to put on their website or social media, or how various groups of people feel about that content. Interpretive scholars are often more concerned about how we think and feel, the unique factors of what drives our behaviors and reaction, than they are about measuring finite patterns and outcomes. For them, the world is full of complex concepts, variables, and variations, and numbers just can’t represent what’s really happening. Critical scholars simply add a layer to empirical and interpretive research by asking why things are happening in relation to the world more broadly. They’re concerned with relationships involving power, ethics, and institutions (including their own power and influence), and are constantly questioning even their own analyses. They wonder why the room is closed off, who is trapped or allowed to go inside, and what it means, not just to them but also to society (or other groups, or the planet, etc.) as a whole.

8  Part I: Get the party started right Choosing a paradigm is important because it allows you to start forming some research questions (RQs). This is useful, even if your questions start off vague. What do you want to know? What questions can be answered by other scholars, and what questions do you specifically need to be able to answer through your research? Part of that means choosing a method, of course, and that method is in part dependent on your paradigm. For instance, most people start hearing about the “scientific method” in elementary school; you Observe, Question, Hypothesize, Experiment, Analyze, and Conclude. This is no different, except we suggest you keep hypotheses to a minimum; focus on the questions. What’s important is how you’re going to do that experiment (method), and this is completely dependent on your resources (time, money, and people to help), perspective (paradigm), and questions. Figure 1.2 gives you an overview of various kinds of methods, and the questions they might try to answer. We’re going to talk you through many of those throughout this book, so don’t freak out. We suggest that before you choose a method, start with some questions. What do you want to know? If you want to know how something is portrayed, discussed in the media, and/or made part of an organization or institution, you might want to choose a text-centered method. You would analyze the message, artifacts, and context that surround them. If you want to see how people respond to something, are influenced by something, or discuss something among themselves, you might want to go with a ­people-centered method. If you want to determine reactions that people might have to a specific stimulus (specific enough that it can be both repeated and contained in a controlled environment), then you’re probably going for an experiment-centered method. Once you know which type of method you’re probably going to use, you can think about number and depth. Because resources are limited, you need to go back to the research questions. What do you want to know? If you need to see how a large number of

• Textual analysis (in-depth reading of one or a few texts, with emphasis on their physical structure, e.g., shooting and editing for film, layout for design, rhetoric for a speech) o Number: low o Depth: high o Sample: intentional • Content analysis (brief reading of a large number of the same sorts of texts, usually with the aid of explicit “coding,” in which commonalities and differences are examined across all texts and charted, usually in part for statistical significance) o Number: medium o Depth: medium o Sample: intentional but systematic; coding is as comprehensive as possible • Discourse analysis (brief reading of a large number of usually different kinds of texts in order to discuss the ideological landscape surrounding a central topic, including journalism, popular media, educational systems, state institutions, family systems, etc.) o Number: high o Depth: low o Sample: Intentional but wide ranging, surrounding a topic/focus • In-depth interviews (structured, semi-structured, or unstructured, interviews are often done with fewer participants but at a high level of depth, usually from

Figure 1.2  Choosing a method

1–2 hours in length, and try to get participants to tell stories and explore concepts in detail, often with ­follow-up questions or examples) o Number: low o Depth: high o Sample: intentional • Focus groups (small groups of individuals that are targeted to gain specific information about a topic; usually random groups or groups that are intended to get a range of opinions, focus groups are widely used in industry studies of media, like whether a new TV show will be popular, or in social sciences to gain perspectives on things like body image, etc.; focus groups are great because you can use a semi-structured interview guide but encourage conversation among individuals, which you record and then analyze; however, they can also result in “groupthink” or silenced voices) o Number: medium o Depth: medium o Sample: intentional • Surveys (a questionnaire given to a sample of individuals to gain their perspective on a topic) o Number: high o Depth: low o Sample: varies from “convenience” to “snowball” to “random” • Ethnography (observation and investigation of the discourses, events, rituals, and communications within

Figure 1.2  Continued

1: Learn to love, not dread, research  11

a community or a (set of) individual(s) within that community; although auto-ethnography is absolutely valid as a method, it must be chosen with great care and attention to the method as appropriate for the sort of impact and evidence necessary for the project) o Number: low o Depth: high o Sample: intentional but wide ranging to encompass many variables • Metasynthesis (brings together a bunch of scholarly sources to examine their results all together, almost like a qualitative content analysis of scholarship) o Number: high o Depth: low o Sample: impartial and wide ranging; comprehensive if possible • Experiment (within a controlled environment, manipulating variables in one group in order to determine changes) o Number: low o Depth: high o Sample: intentional with a control group

Figure 1.2  Continued

people think about or react to something, you need a high N (the population size being analyzed). You’ll probably need a survey in that case. If you are more concerned with the nuances of how people think or feel about something, you will need them to tell stories or discuss the topic in detail with you. In that case you

12  Part I: Get the party started right might want to do interviews with a high D (depth) that could take several hours each. If you’re shooting for something in the middle, maybe how a group of people will respond to a new advertisement or feel about their romantic partner one year after getting married, then focus groups are better bets. The same thing goes for text-based and experiment-based methods; in general, the higher the N, the lower the D (see Figure 1.2). That doesn’t mean that high-N methods don’t have depth. It just means that with limited resources you’ll have to focus on one over the other. Be careful, when choosing the sample that you’re going to study (which texts, which people, which places, etc.), to avoid what we call “cherry-picking.” Cherry-picking means limiting your sample to the sources that will support your argument, which can be avoided by focusing on questions rather than answers. For instance, one of Ted’s students had a problem last semester with studying body image on Instagram. A good topic, certainly, but when she tried to study it through a content analysis, she pulled images from celebrities and focused on their “sexy” photos. What if some or all of those celebrities posted just a few sexy photos, but more often posted pictures of normal life: without makeup and in ugly pajamas, pictures of their pets and kids, and random pictures of food? Her research would be skewed because she just chose primary sources that suited what she thought were going to be the answers to her questions, rather than letting the data speak for itself. The thing to remember is that you don’t have to do any of this alone. Other scholars have struggled with finding a topic, creating research questions, and choosing methods since . . . well, since people started doing research. And because analyzing the world around us is fundamental to the human experience, if not life itself, that’s a long, long time. Reading other scholars is one way to keep yourself from getting overwhelmed. But so is talking to people, including your friends, your family, your professor, your colleagues, your boss, and of course, your friendly neighborhood librarian.

1: Learn to love, not dread, research  13

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: Librarians know that your first (or tenth!) research project can be a really daunting process. Every semester, Maridath works with dozens of students who are overwhelmed by even beginning their research assignments. Often, your professor will spend a great deal of time explaining the requirements, but they often forget how hard it is to come up with a good topic. We can help! Unlike the boss that wants to know everything about “how satisfied the customers are,” we want to help you both with the broad questions and with how to do the research without driving yourself crazy. We take you through the process step by step, from getting to know you and your interests, to creating and focusing a topic and the research questions suitable to your project, to actually finding the resources to make sure you’re successful. We’re great at helping you find sources for your research assignments, but we really enjoy the process of helping you find a topic that’s interesting to you so you won’t get bored and hate us and your professor (or boss) later. You may not realize that we also work closely with your faculty and often have the inside scoop on what they’re looking for. It’s important to also come to your session with your librarian with any work you’ve already completed on the assignment, including preliminary sources, ideas around your topic and focus, and, of course, the assignment prompt itself.

CHAPTER

2

Where do I start?

OVERVIEW AND OBJECTIVES This chapter teaches you how to effectively and efficiently retrieve and use scholarly sources for your project. This process allows you to draw on existing research, making your job a lot easier. It also avails you of the support of those who have been working on this stuff for a long time, often in ways that you aren’t going to work on it. It always helps to have multiple perspectives. We give you an idea of what a good source is, but also what a bad source might be, how to avoid them, and some tips and tricks for how to find good sources without driving yourself nuts. Finally, we talk about how to create a good annotated bibliography to keep yourself organized and on the right track.

2: Where do I start?  15

Now that you have a topic that you don’t hate and that you can feasibly accomplish, when do you get to the actual really real research stuff? To be honest, you’ve already started. Choosing the topic, asking the questions, and finding people to help you are part of the research. In fact, they are the most important parts. Everything else is comparatively easy, even if it’s hard work (yes, we know that sounds weird). Your librarian in particular can help you here, because librarians are excellent at helping you find stuff out, no matter what “stuff” means. They don’t do that through a research method the way we’re describing it; instead, they help you by giving you a step up. To use a common saying in English, they help you “stand on the shoulders of giants.” So what does that mean, realistically? It means librarians help you find other people who have tried to answer similar questions, or find ways in which other people have studied similar topics, so that you don’t try to do everything (or that you don’t waste time doing something that someone else has already done). The trick is that you need to be able to keep this process methodical. If you just start randomly choosing articles or books that have to do with your topic, you can run into some major problems.

Problem 1: Bad sources The main reason you might find “bad” sources is if they don’t pass the “CRAP” test for a source’s “currency” (how recent it is), “relevance” (how related to your topic and its particular discipline), “authority” (how credible the publisher or writer is, and if they have authority in that particular field), and “purpose” (are they just trying to persuade you of their opinion, or do they want to inform you?). In general, for secondary sources you’ll want to stick with scholarly, peer-reviewed journals. In some cases, like with the project on nonprofit organizations, it can also be important to get sources from established “trade” or

16  Part I: Get the party started right “industry” publications. These might give you current perspectives on what works or doesn’t work in an industry or practice (that could be website design, but it could also be doctor-­patient interactions or adapting a novel to the theater). Even major newspapers like the New York Times or the Washington Post (if they aren’t editorials) can be useful, but use them sparingly.

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: What’s peer review and why should I care? So who are these “peers?” And what exactly are they reviewing? Librarians love to talk about “scholarship as conversation” – seriously, get one going on this for a minute and see how excited they get. Articles that appear in peer-reviewed academic journals are written by experts for experts. After performing their research, scholars submit their work to these journals, which then undergo a review process. Their work is checked for style, substance, and quality by other scholars in the field, usually in a double blind method where no one knows the identity of anyone else. Articles and books are then either accepted for publication in the journal, accepted with revisions (where the scholar must make often substantial changes and then re-submit), or outright rejected. Usually, from start to finish, this process can take a long time – sometimes years. It’s important to note the length of this process, because this notion of “currency” (the “C” in the CRAP test above) really differs between

2: Where do I start?  17

scholarly journals and newspapers; it also depends on discipline. If you are researching something that happened in the news last year, there might not be much in the scholarly literature on it. What you will find is an ongoing discussion of sorts – scholars citing and commenting through time around the overarching themes and theories of their particular discipline – thus, the scholarly conversation on, say, the news industry will be more likely than something on a specific current event. Newspaper articles also go through review, but usually by an editorial staff (if that), and rarely by people who know anything about the content or broader facts/context. Journalism is in many ways “normal” people speaking to other “normal” people in an easily understandable way, and they often interview scholars for the context they need. Ted, for instance, goes on the news often to discuss culture, politics, social media, and communication in general (his first appearance was for a spot on online dating, which he’s still embarrassed about). It’s important to note that scholarly sources are not necessarily “better” or “worse” than newspaper articles or more popular texts, but they serve different purposes and wildly different audiences.

Problem 2: Out-of-date or irrelevant sources While this depends on what your project is and who has assigned it (your boss, your professor, yourself), usually you want to make sure your sources are current and useful. While not necessarily “bad,” you might find that out-of-date sources are easier to find because they are “kind of” about your topic, but not really useful

18  Part I: Get the party started right in the end. Sources that are “kind of” about your topic probably don’t pass the “R” (relevance) part of the CRAP test. In some cases, older sources may be appropriate (e.g., if you need to trace the origin of a problem or track its historical trajectory). That said, it’s perfectly fine to consult sources that inform you about the topic more generally. Just be careful to keep them to a minimum. For Ted’s project on disability and theater, for instance, it was useful to look at sources on mental disabilities in the media in general, and ageing in the media in general. However, looking at medical studies on autism or on Alzheimer’s probably wouldn’t be useful for a paper on how it’s portrayed in the theater, or how audiences react to those portrayals. Similarly, be sensitive to how old a source is. Old doesn’t mean bad, but it does mean that the author(s) weren’t able to react to conversations that happened since. Remember, scholarship is a constantly evolving discussion.

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: But are Google and Wikipedia BAD? I thought they were BAD? Many researchers choose to begin their research on Google Scholar (not Google!), or even Wikipedia. We like to do this because Google Scholar allows us to do a couple of unique things: see the impact factor or how many times a particular journal article is cited by other researchers, and quickly find related articles on a particular subject. Often, Google Scholar is indexed in your library’s main catalog via complex metadata sharing which will allow

2: Where do I start?  19

you to follow the link to the full-text PDF of the article. A word of caution, however: you can usually only do this from a computer on campus. You should never, under any circumstances, pay the fee for a source you find on Google Scholar. What you should do (especially when off-campus), is take the title and/or author’s name and plug it into your library’s main search bar and see if they provide access. If they don’t, you can often request that the source be sent to you by “Interlibrary Loan,” or ILL, which often means a librarian from another university will scan the article and email it to you, or even send you a book in the mail. As you progress further into your research and your topic narrows, it’s a good idea to look through the more discipline-specific databases to which your library provides access, but starting either at Google Scholar or the library search engine is the best place to begin. Google Scholar also has a wonderful set of tools to narrow down your search, including “related sources” and “cited by,” which will allow you to find useful material that you might not see in a regular search. Similarly, Wikipedia allows you to get a sense of what people have said about a topic’s history, background on associated content (e.g., the filmography of a director whose film you’re analyzing, or the biography of a historical figure whose speech you’re comparing to national policy), and even news press or scholarly sources. Wikipedia can also be useful for giving you a primer on the terminology associated with your particular topic, which can aid you in crafting quality search terms. Like with other source

20  Part I: Get the party started right material, make sure you look for citations and give those the CRAP test!

Problem 3: Cherry-picking “Cherry-picking” is a pitfall of one’s method and use of primary sources, as we mentioned above. It’s also important to avoid in your literature review. In the secondary sources for your annotated bibliography and literature review, cherry-picking means that you’re just choosing sources that agree with you. Don’t do that. The more diversity you find, the better. The “literature” isn’t monolithic. Instead it’s a series of people just like you (maybe with a few more letters after their names) that are asking similar questions, and answering them in many ways. Often, that means they disagree, and that disagreement is important for your research. One of the best ways to keep these problems under control is to make sure that you’re searching for things strategically; but you should also make sure to constantly critique the sources. Keep in mind the notion of authority as well. Ted has expertise in several fields, but medicine isn’t one of them. Medical journals on autism weren’t exactly the best for his research on autism and theater, and he shouldn’t make medical claims when he writes. In other words, don’t get overwhelmed by the fact that an author has a bunch of degrees; you should still be critical of their work. Similarly, you should keep track of how you’re searching and what you’ve found, even if it’s irrelevant stuff. After all, if you have to come back to this topic you’ll want to know what not to look at, and if you’re handing the project to someone else (another student, an assistant, a professor, a boss, etc.) you want to make sure they don’t have to do all that work over again either.

2: Where do I start?  21

IRL: RESEARCH CAN GET YOU PROMOTED, BUT MAKE SURE YOU DO IT RIGHT One of the most common pieces of feedback we get from students is how surprised they were when the research they did in our classes is the skill they use most often in work (and what often gets them better jobs). It doesn’t matter if we taught them to analyze scenes from a film, conduct surveys, or interview their peers; the process of choosing and conducting high quality research is marketable and translates across industries. Our students conduct focus groups and interviews with employees and customers to figure out how to make their experiences with a company better. They create surveys to measure how professionals work most effectively, how doctors perceive their patients or their pharmaceutical representatives, and how donors and volunteers relate to a nonprofit. They compare the social media used by competitor brands in content analyses, and examine that along with analytics to see what works to achieve the company’s goals and what doesn’t. They do textual analyses of their company’s advertisements or press releases to avoid public relations crises or unintended messages. But what connects all of these is that they have to do it right. Making mistakes like ­cherry-picking in a sample, ignoring key research, or using bad sources might not make you fail a class, but it can cost your company millions of dollars. If you do it right, on the other hand, the sky’s the limit.

22  Part I: Get the party started right Keeping a consistently formatted bibliography of sources will help you a lot with the project. People complain about this all the time, including us. It always feels like it slows down the process, because when you start your research you want to go quickly, and nothing feels slower than formatting sources. However, trust us that by the end you’ll be glad you did. Every minute you spend here might save you two or three (or more) later down the line, usually when you’re pressed for time. We create bibliographies with consistent citation styles such as APA, MLA, and Chicago so that we can organize our sources consistently and methodically. Why does this matter? First, you save time by ensuring that you and your readers (e.g., your professor) can quickly and easily find the sources you are referencing, or specific pieces of the source like the journal or the year it was published. You also prevent plagiarism or a dreaded visit to your school’s Honor Council, which could ruin anyone’s day. But the main issue is that this is useful as you gather more and more sources, and especially if you’re doing this for work, when they might be also used by your boss or a team. For most books, for instance, you consult not dozens, but hundreds of scholarly sources. Without keeping a bibliography, you not only risk forgetting the sources but also leave out key pieces of information and background.

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: Citation exporters There are many ways to cite sources, but they’re all systematic. You’ll most likely encounter the styles from the Associated Press, the American Psychological Association, the

2: Where do I start?  23

Chicago Manual of Style, and the Modern Language Association. You’re probably aware of a few tools that allow you to select a citation style and click a button to “export” the citation into a word document. These have names like EasyBibb, EndNote, Zotero, or NoodleTools, some of which are free and some not. Zotero is one of Ted’s favorite tools, not because it formats things correctly, but because you can add it to Microsoft Word and to browsers like Google Chrome. It lets you grab citations from articles and even websites and newspaper articles quickly as you find them, and then go back to download or read them later. There is often a citation exporter button in the search results page on your library’s website, too. Zotero is free and usually works the best, so it’s the reference management tool we recommend. Any of these tools will get you most of the way there, meaning they’ll help you capture the elements that are required in a properly formatted citation. But the reality is that they often make mistakes. Use them, but fix what they produce! Many sources, like the official citation guides from each of the major style manuals, make this an easy process (especially for formats like Chicago that have two different versions). One other suggestion is to make sure you keep things organized, and on the “cloud.” Using OneDrive, Google Drive, or iCloud, keep your sources (like PDFs with highlights) and every element from your paper stored off your computer, so you don’t risk losing all your work. Trust us. We’re all still traumatized by losing huge amounts of work when computers crashed or we forgot to save a file before the cloud was there to save us from working after too little sleep and too much coffee.

24  Part I: Get the party started right

Annotated bibliographies You also want to make sure that you don’t choose some sources, read them, and then forget what you read and have to read it all over again in a week (or a month, or a year). The best way to prevent that is to create an “annotated bibliography,” which is a bibliography to which you’ve added notes about each source (including whether it was useful or a waste of time to read). What we recommend is keeping this organized for yourself, either with bullets under each source or in a short paragraph. While there are various way to do this, there are a few things you want to make sure to hit, some of which we discussed in Figures 1.1 and 1.2: 1. Topic (what’s their general concept?): Don’t try to get too specific here. Think general search terms for Google Scholar like “gender and advertising” or “disability and media,” or even “doctor patient communication.” 2. Focus (what specifically are they looking at?): This is where you will want to be more specific about what you’re studying. Is it a set of places (hospital waiting rooms, elementary school architecture), texts (poems, films, speeches), people (patients, millennials, students), concepts (deception, humor)? How specific you want to get depends on how far along you are in the project. Eventually “humor” probably won’t be enough. At some point, for instance, you’d study how college students respond to humor in a specific episode of The Office regarding the intersections of class, race, and gender. 3. Method (how did they study it?): This varies, but you’ll want to specify sample (and sampling technique) as well as the general method (see Figure 1.1). 4. Results (what did they find out?): Connecting your method to your focus, how did the study answer its research questions? Be aware that the questions might not be obviously laid out, even in a thesis statement. You might have to look for them.

2: Where do I start?  25

5. Implications (why does it matter?): Implications are both your judgment about the source’s relevance to your project and also how the author(s) argue the study is important. Often this is in a “discussion” section, but usually it’s towards the end. Limitations are also a good thing to include here: what should happen next? Where did the study fall short? For example, in Ted’s paper on disability in theater (topic), he examined the portrayal of cognitive impairments in two plays (focus) through a textual analysis of two scenes each (method). He found that the plays emphasized difference through a mixture of disconnection from society and empathy with the cognitively “different” characters that encouraged viewers to be critical not of the impaired, but rather of the “normal” characters and an inaccessible society (results). He suggested that media is thus a potentially effective way to broaden social recognition of a cognitively diverse society, reducing stigma and encouraging empathy even if its portrayals are fictionalized (implications). Once you have your sources laid out, categorize them and ask yourself two questions: First, have you addressed all of the aspects of your topic that are necessary for your project, and in enough detail (with enough sources)? Second, what sources did the studies that you read cite that you might be able to use? Once you’ve got that figured out, you’re ready to start your literature review. This is also why you want to spend as much time as you can on the annotated bibliography; if you start with the literature review, you might end up spending a lot of time on sources you don’t really need, or you might miss out on more helpful sources. More importantly, it’s harder to categorize and see themes when you’re immersed in the sources. The annotations help you take a step back and see the big picture.

CHAPTER

3

Now what do I do? What’s a lit review?

OVERVIEW AND OBJECTIVES This chapter takes you through the process of making a disconnected set of sources into a coherent, persuasive “literature review.” Basically, that means you’re going to tell the story of your project, through the scholars who have written on the same or similar topics before you. We provide some suggestions for how to find categories of literature to look for before writing the literature review, which involves drawing from and expanding on your annotated bibliography. This is sometimes a tricky process, so we’ll take you through how each of us approaches it in detail to give you some guidance on process.

3: Now what do I do? What’s a lit review?  27

Once your annotated bibliography is done, it might seem like you’re ready to dive into your paper. In many ways, you’re right! This is where the project truly begins to take shape, and it’s important that this happens before you ever start to do your analysis, or even before you choose a method. That might seem strange. After all, what is research if you aren’t actually doing anything? Why aren’t we trying to answer the questions we spent all that effort framing? The answer is simple, really. We don’t know enough yet. We might have an idea of what we want to talk about, and we might have thought about it a lot. We might have even read a bunch of scholars talking about our topic, and so we know its history and how people have approached it over the years. But if you start writing now, you’re going to run into problems. Why? Because the first part of writing the paper is making sure you know what’s going on, categorizing it for yourself and, more importantly, contextualizing it for your readers. It’s important that they know that you know what you’re doing.

IRL: WHY WOULD I EVER HAVE TO DO THIS AFTER I GRADUATE? It might seem like a literature review is one of those things you just do because your professor makes you. You’re not wrong. But that doesn’t mean you won’t have to do it in some way after you graduate, or that it’s not a useful skill that will be valued by your company (and might even get you promoted). The reality is that while you’ll rarely do a traditional literature review in a job, you’ll almost definitely have to synthesize complex information from a bunch of sources that might not be directly about your project,

28  Part I: Get the party started right

compare various authorities, and insert yourself into a conversation that you aren’t yet a part of. That’s really what it means to be a manager. Knowing how to do a literature review means you know how to see what you’re doing as an extension of what’s already been done. It means you can look at what competitors and people that disagree with your perspective are saying, and take what’s valuable from them before respectfully suggesting that there’s another solution. Imagine your company is merging with another company, for instance. What are the different ways that’s been done? What causes employees to stay happy and become hopeful rather than fear the change? What causes shareholders to increase the stock price, customers to keep buying the products your company creates, and the research teams to work together? There are differing views on all of these things, and each is a different research question. Knowing how people have approached the problem, as well as what worked and why, will help you suggest and provide a solution that is grounded in research and has some backing when colleagues or bosses are skeptical or upset. For instance, Ted once told a client who was worried about low profits that their biggest problem wasn’t their employees, who were blamed because they weren’t looking for new business. Instead, he said it was the bosses that had the problem, and every employee that left not only cost them money, but scared away potentially great new hires. They didn’t want to believe it, but when they saw the data and the literature on the topic, they started to rethink how they approached all sorts of things, from meetings to salaries to benefits. It worked; but he couldn’t have done it without the research.

3: Now what do I do? What’s a lit review?  29

This is where the literature review (or “lit review”) comes in. A literature review is pretty much what it sounds like: it’s a review of the literature on a topic. But this isn’t literature like “English literature.” It’s the scholarly literature, the conversation around your topic in its various forms. How deep you want to go here depends on your topic. But there are a couple of things that can guide you along the way other than a professor saying “you need ten sources” or something along those lines. We do that to our students, don’t get us wrong. However, we don’t care so much about the number as much as we do about getting you to expand beyond cherry-picking your sources and just going along with what they say. Here are some ways you can approach the lit review that might be helpful: 1. It’s about the conversation You’ll never be able to read everything ever written about your topic. If you’ve chosen it right, your topic is broad enough that lots of people have already written about it in some way. Be careful about limiting yourself to your more narrow focus, even if there’s been a lot written on it, because that conversation is only part of a much larger one. In some cases, there will be tens of thousands of pages written on your topic. Every semester we hear students say “but no one has written about this!” or “but I can’t find anything on my topic!” They’re always wrong; they just don’t know what to look for. So here’s the thing. Your topic isn’t a set amount of information. It’s a conversation about an issue. That means it’s a bunch of people thinking about, studying, and writing about something connected to what you’re interested in. The hottest new thing in music, technology, or science is never really new. So you just have to find out who’s talking about it and listen in. It’s kind

30  Part I: Get the party started right of like going to a party where you don’t know anyone, and wandering around until you find a group that’s speaking about something interesting. You probably shouldn’t just jump in and start speaking, even if you’re an expert. You have to ease into it. Think about this in terms of music. When you listen to music, you aren’t just listening to the artist themselves; you’re listening to the people that influenced them over time. It’s one thing to enjoy music, and another to be able to appreciate it through its history. To truly understand what’s going on with Kendricks’ “Damn,” you might need to look at how similar emcees have dealt with issues of race, power, and money. So that might be Jay-Z with “99 Problems,” Biggie Smalls with “Mo’ Money Mo’ Problems,” or Wu Tang Clan with “Cream.” You could also look at lyrical style and the use of musical instruments, tracing the artists back through early rap, jazz, blues, and so on. At the same time, you don’t have to wait forever, because there’s always more to find. Every project is like a rabbit hole that has no end. What you want to do is get a feel for what various people are saying and what the pieces of the conversation are before you can contribute in a meaningful way. While that might be enough for many projects, especially as a student, when you’re trying to do a more comprehensive job it might require what we call “saturation.” Saturation basically means that you’re not hearing anything new, and everything someone talks about has already been discussed by someone else in some way. Practically, this means that most of the citations in any article you read reference other articles you’ve read. Eventually, it will seem like everyone is just writing about (and to) each other. And that’s exactly what they’re doing. Welcome to saturation.

3: Now what do I do? What’s a lit review?  31

2. It’s rarely just one conversation When you want to understand how something works, it’s not enough to listen to just one person, or even a few people. If everyone’s saying the same thing, you’ve reached saturation, right? Maybe . . . but maybe you’ve only looked at it from one point of view, like talking about politics to a bunch of people who think like you do. In one of Ted’s pieces about customer experience, for instance, his literature review went through several ways scholars characterized and measured how customers interact with a brand (and whether or not they’ll become loyal customers). Not only is there disagreement about this, but often the various perspectives seem like they don’t know each other at all. If he had only looked at one, he never would have been able to do his project well. In fact, while his paper was on marketing, it was engineering that gave him his breakthrough. It’s like the blind men and the elephant parable: five blind men touch different parts of an elephant, and each thinks they know what it is (a snake for the trunk, a tree for the leg, etc.). 3. It’s all in how you organize it Here’s the thing. Even if you reach saturation, you’re still going to be overwhelmed. Why? Because sometimes you have thousands of pages of material connected to your topic, and you often only have five pages in which to discuss it. So what do you do? You categorize. You look for themes (groups of people at the party) and the perspectives within those themes (the variety of conversation). You order it in a way that makes sense as a story that explains your topic and what you think is going on. And you explain not only why it’s interesting and important, but also how your project fits in. In other words, you’re taking it to the next level.

32  Part I: Get the party started right So you’ll want to make sure you address not just the general concepts of what other scholars have said, but also in many cases their methods and results. You don’t have to talk about everyone; one or two examples from each theme will usually suffice, and you can always just cite additional examples without discussing them in detail. 4. The lit review is the opening act. Your research is the main stage Imagine you’re at a concert (music or a stand-up comedian or whatever). Before you see the act you paid to see, you’ll almost always see an “opening act” or two. They get you warmed up and in the mood to enjoy and appreciate the main act. After all, if you’re a comedian, the last thing you want to do is make jokes to people who’ve just gotten off work and are tired or not in the mood to laugh. The literature review does just that. It sets the stage for your own research, showing why it’s important and what you’re doing that’s interesting, new, and different. But if you don’t have that opening act, it’s not going to make much sense. The biggest challenge of doing a literature review isn’t why, but how. Even once you have determined your categories, it often seems overwhelming, especially if you have a lot of sources. There are many ways to approach the problem, and we’re going to give you a couple here. Everyone has their own process, but these have worked for us. The thing to remember is that this process will lead you to making your argument. How? Well, once you’ve established the literature, you’ve made a clear argument about what’s been done, as well as what’s missing or incomplete. So that allows you to bring in your method and your choice of data as a way to help complete the puzzle, adding yourself to the conversation.

Politics and identity in Zootopia (2016) and Moana (2016) I. Race, authenticity, and politics in animated film II. Disney and identity A. Princess vs. anti-princess films III. Zootopia and Moana Fast fashion and environmentalism I. Advertising and fashion II. Fashion and unsustainability III. Fashion and environmentalism

Identity in Orange is the New Black I. Portrayal of criminal justice in the media (vs. reality) II. Portrayal of women in the CJS III. Orange is the New Black Images of plastic and pollution   I. Activist photojournalism II. Activism and social media III. Environmental activism

Re-thinking James Bond I. White masculinity and the media II. James bond and identity III. Evolution of a franchise and identity shifts

Adaptations, power, and The Blind Side   I. Football films II. “White savior” films The Blind Side III. Adaptations of “true” stories

Identity and The Last Jedi (2017) I. Sci-fi, politics, identity II. Star Wars, politics, identity III. Evolution of a franchise and identity shifts

The Office and mockumentary TV I. Portrayals of work and office life in the media   II. Mockumentary genre III. The Office series (US and UK)

Figure 3.1  From topic to literature review (samples from popular culture and society)

Monogamy, polyamory, and the media I. Monogamy constructions and discourse, data on relationship satisfaction and longevity II. Individual perceptions of monogamy vs. polyamory III. Monogamy in the media IV. Polyamory in the media

Women’s athletic swimsuits I. Gender in advertising II. Perception and portrayal of female athletes III. Perception and portrayal of female swimmers (maybe) IV. Possible impact of advertising on women, including athletes (maybe)

Instagram and humor I. User-generated social media theory (Instagram, video, memes, etc.) II. Humor and social media III. Theories of humor

Lacrosse stereotypes I. Stereotypes of athletes A. Gender B. Sport II. Portrayal of athletes in the media

Female celebrity identity on Instagram I. Celebrity branding and gender/ sexuality II. Celebrity and social media III. Parasocial relationships and aspirant identity IV. Social media and self-portrayal V. Social media and self-perception

(Mis)representation of Africa in US media I. Representations of Africa in the media A. Positive over time B. Negative over time II. Black Panther or other Africafocused films from Black African perspectives

Anti-Photoshop discourse and legislation I. Advertising, gender, and body image II. Photoshop and/or image alteration III. Anti-alteration legislation and “natural” body portrayal movements

Superman and religious iconography I. Portrayals of superman II. Superman and other comic book icons as cultural/political allegory III. Superman and religion

Figure 3.1  Continued

Revenge and identity in Insatiable (2018) I. Teen-targeted media, narrative, and identity II. Revenge in the media III. Body image and the media (obesity-focus)

Representations of women in the Harry Potter series I. Gender representations in children’s or teen literature II. Harry Potter series III. Gender in Harry Potter series

Social media addiction I. Addictive behaviors and treatments II. Social media addiction (psychology focus on individuals) III. Social media use (trends and addiction) (communication focus on media)

Gender and cyberpunk 2020 I. Sci-fi, identity, and gender   II. Role-playing games III. Cyberpunk

3: Now what do I do? What’s a lit review?  35

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: It’s worth it. Trust me At this point, you might be struggling to understand the difference between an annotated bibliography and a literature review, and I get it. It seems repetitive and maybe even like busy work. It certainly can feel redundant. What really sets a lit review apart from an annotated bibliography, though, is that this is where you begin your narrative. Remember, every piece of writing is persuasive, and this is when you go from reading about others to framing your own argument (even though it’s through other scholars at this point). Your annotated bibliography simply summarizes your sources thus far, keeps them organized, and explains why that particular source relates to your topic. Your literature review is where you make decisions about which topics you’re going to keep, and begin exploring the themes that exist between your sources. It will be in paragraph form, because if it’s done correctly it won’t just save you time writing your actual paper. This is actually the beginning of your paper! You also don’t have to reinvent the wheel – there’s usually tons of online guides that your librarians have created, called LibGuides, which can help with finding resources and how to complete specific assignments. Check out our online resources for some sample LibGuides.

36  Part I: Get the party started right

Ted’s method The first thing I do is download a bunch of articles, and save the citations to Zotero. By “a bunch of” I usually mean at least thirty, sometimes more. My favorite tool for that is Google Scholar, because the links to full text (at least, when I’m on campus) are right there next to the article and I can download them quickly. The “cited by” and “related to” links are also amazing. Once I have a big set of articles, I rename the files like this: AuthorLastName-Date-Topic.pdf, and although that might sound like a pain in the butt (and it is), it’ll save you tons of time later on. I put them in a file folder that is automatically synced to Microsoft’s OneDrive so I don’t lose anything, and then I start to read. But here’s the thing: I don’t read everything right away. I read the abstracts for the articles and then prioritize them. Usually that starts with sub-folders. In my disability piece, I made a sub-folder for ageing in the media, one for autism in the media, and one for theater and activism. Then I chose the articles I thought would be best out of that list (I actually made a partial annotated bibliography from the abstracts, and highlighted those articles to prioritize in the document). I read those articles first, highlighting them within Adobe Acrobat, and then put the highlighted articles in a “read and highlighted” folder to keep myself organized. I also looked up and downloaded any articles they cited that looked useful or interesting; it wasn’t long before those sources all started to refer to each other, which meant I was reaching saturation. This is the point where it often seems overwhelming for people, and here’s how I dealt with that. I took all the highlights and comments I typed into the PDFs and copied/pasted them in a new document for each folder (ageing and media, for instance), with an in-text citation (e.g., Gournelos 2009, 56) next to each

3: Now what do I do? What’s a lit review?  37

one so I could find it again. This created three very, very long documents. Suddenly it wasn’t so much about what to write, but what to cut; I had five to eight pages that I could use for this at most, and I had fifty pages or so of quotations alone. So what did I do? I highlighted them again, in different colors depending on what part of the literature review (what theme) they fit. Then I organized them by color/theme and read through them all together, putting quotations that were similar together and reducing quotations by paraphrasing where it made sense. I outlined the literature review with those sub-themes as headings, and then started to write, cutting/citing/quoting from my sources as I went. Fifty pages turned into ten, as the literature seemed to just write itself, logically and in an order that made sense. I was telling a story, in other words, drawing from the stories of other people. Then I just went through and edited, usually removing quotations or pieces that I thought were going to be relevant but really weren’t, or needed to be one sentence instead of an entire paragraph. It helps to remember that you’re trying to answer a question, and sometimes there’s some great, interesting research that gives you context but doesn’t help you with your question. You can usually reduce or cut all that stuff. That was it! End of literature review. I’ve used this for everything from a proposal for a book to a journal article to a full, three-hundred-page book. It works, and it keeps you sane.

Josh’s method Once I knew what I wanted to study – best practices in healthcare communication − it was time to find out what everyone else has already said about my topic. Unlike Ted, I start with the library’s databases, usually LexisNexis or EbscoHost. Depending on how vague my search terms are, I can pull up anywhere from twenty to twenty thousand articles; if necessary, I narrow

38  Part I: Get the party started right that with phrases that specify my context (e.g., Physician Communication Patient Satisfaction). Once I have a manageable number of articles to choose from, I begin by reading their abstracts. When I confirm that the article’s goals and findings align well with my topic, I save them to a folder in the database. I’m too tactile to organize the literature review on my computer screen like Ted does. Call me old-fashioned, but I print out the twenty-plus articles that I think are the best. I take a pad of sticky notes, and I write keyword descriptions on each and stick them to the top of the front page of the article. One article might have five post it notes stuck to it: (a) Trust, (b) OpenEnded Questions, (c) Empathy, (d) Nonverbal Behavior, and (e) Social Support – as variables that relate to Physician-Patient Communication. The other nineteen articles may have a similar number of sticky notes, some with recurring themes, others distinct. I then create piles by placing articles with similar themes together. At the end, I may have three major sections in my literature review: (a) Physician Nonverbal Behaviors, (b) Physician Verbal Behaviors, and (c) Physician Characteristics. Now that the “conversation” is organized, I can start writing. I always try to remember that my audience doesn’t need to know everything. I ask myself, “What in these articles do they need to know to help understand the conversation before I add my voice?” Usually that means I include the basic (a) goal of the study, (b) significant findings they mention (hypothesis supported? answers to research questions) and (c) the holes/missing pieces to the puzzle that my study will address. I then go through each article and highlight the study’s (a) goals, (b) major findings, and (c) holes, then rephrase them in my own voice and cite them. The first draft of my literature review is always choppy and reads like a bulleted list of goals and findings from twenty odd sources – because, well, it is. Next, I go through the literature review again, adding preview statements at the beginning of each theme, combining similar studies together, and transitioning

3: Now what do I do? What’s a lit review?  39

between themes to create a better flow. By the end of the literature review the reader should know exactly what you hope to accomplish with your study. A great former professor once told me: You’ll know you’ve written a great literature review when the reader knows exactly what the hypothesis and/or research questions are before they ever read them.

Maridath’s method The research that I’m engaged in as an information science scholar is a little different than Josh’s and Ted’s, but the literature review method is pretty similar. In a recent project on librarians who teach information literacy/research methods courses, I narrowed my focus after a quick look through the databases to using “backwards course design” as a way to create effective courses. It was through my early reading on my topic, when it was still at its broadest stage, that I learned which part of my topic I was most interested in. What I learned was that when I redesigned the syllabus for my course, I did it beginning with what I wanted the students to know by the end of the course rather than starting with what I thought they should learn from the beginning. I learned that this had a name: backwards course design. It was through my initial reading that my topic gained a focus, and I was able to narrow my research as I made my way towards the annotated bibliography and literature review. This is different from the kind of original research we talking about later in the methods chapters of this book, except you might consider it a kind of theory research (see Chapter 9). My method falls somewhere between Ted and Josh’s – I begin with searching a Library & Information Science-specific database, because my field is so niche and specific. The one that I prefer to use is called LISTA – Library, Information Science, and Technology Abstracts with Full Text. You’ll note by the

40  Part I: Get the party started right name that some of the content I find in this database allows me to get to the full-text PDF of the articles, but some of it only provides the abstract, at which point I use Interlibrary Loan (ILL). After I find roughly twenty full-text articles (usually each open in different tabs), I use the citation exporter located within the database to paste the citations in APA format into a Google Doc (I fix them later). Then, I quickly read all of their abstracts to see if I’m on the right track with my search terms. For any of the articles that seem worth the time, I either request them via ILL or read them in their entirety if I have full-text access. Usually, if I’m lucky, at least half seem good enough at this broad stage, and so I then print each one. I like the feeling of actually holding the articles in my hands, and truthfully I’m less likely to be distracted by my computer and can focus better if I read them this way. Unlike Josh, I prefer not to save the articles in their respective databases, probably because I work so closely with electronic resources and know that often these links sadly suffer from code rot and can disappear. In other words, you’re better off capturing citations at minimum or complete PDFs at best. After reading each article, I write several keywords on each that will help categorize and group them by themes. For this project it was themes such as: “backwards design,” “information literacy,” “course redesign,” “threshold concepts,” and “metaliteracy.” I group these articles into piles based on their themes, and for articles that strongly address more than one theme I decide which seems to be the most important to the author(s) and place them in that pile. For each pile I then go through each article, fleshing out the annotated bibliography in Google Docs, remembering to check my citation format at this stage with a citation guide. I also use a highlighter to pluck out quotations in the articles that I think will be useful for the literature review. Next, it’s time to take my Google Doc of annotations and summaries and put them into order by theme, based on the piles of articles I’ve created.

3: Now what do I do? What’s a lit review?  41

As with Josh’s method, at this stage my annotated bibliography really looks like a list. From here, I work to turn each summary into complete sentences, quoting excerpts from the articles where it makes sense. Each article is grouped by theme, and these themes become sections of my literature review.

CHAPTER

4

Choosing a method

OVERVIEW AND OBJECTIVES This chapter will show you how to move from a literature review to the method you’ll use for your analysis (or multiple methods if you want to get fancy). We’ll show you how to draw from the literature to choose your method logically, and what questions might be answered with each. We break methods down into two main pieces: “portrayal” (how things are constructed or shown) and “perception/reception” (how things are interpreted and viewed). Because ethics are a big deal with methods, we have included “ethics alerts” throughout that will tell you what to watch out for and why. At the end we’ll give you an idea of how various methods could be used in the business world.

4: Choosing a method  43

Once your literature review is done, you should have a clear idea of what others before you have argued about your topic and through what methods. At that point you have four choices for your own method: 1. Precisely duplicate what’s been done before to support earlier arguments if you think that’s necessary, or contradict it if you think there are some serious shortcomings. 2. Duplicate what’s been done before, but with a new context (a different TV show, a different country, a different demographic or psychographic of participant). 3. Duplicate what’s been done before, but with a new method. 4. Go out on a limb and argue in a different way using a different method. We don’t really recommend #1 or #4, because in most cases #1 will be boring and won’t really expand the field significantly, and #4 is very risky. Usually you do #4 if little has been done about your topic, and there’s not much that’s relevant in the broader literature review. That’s very, very rare, though; in fact, it probably just means you didn’t explore the literature expansively enough. Take a look at that first; if you can stand on someone else’s shoulders, it’s a lot easier than jumping up and down to see over a crowd. No matter what, though, you’re going to have to choose a method, and you’re going to want to do this in a way that can be clearly explained in the paper and that can be easily repeated if necessary. There are a lot of methods for research, and even more varieties of each method. We’re going to walk you through a few of them, but before we get to those chapters we wanted to give you a feeling for what separates them (it’s all about organizing into themes, remember?). We’ll use a central example so that you can clearly see the differences, which is a project Ted is working on with a student regarding the Netflix series 13 Reasons Why. Figure 4.1 also provides you with a set of methods

Close examination of two scenes from two plays about disability (one about autism and one about dementia), dwelling on issues of narrative style, imagery, and tone, as well as place within the story.

Analysis of trailers or advertisements of twenty films and television shows featuring a character with a disability, particularly autism and dementia, coding for issues of representation and focus/erasure of disability in society.

Discussion of the concept “disability” as it appears or is erased/concealed in critic reviews of and newspaper articles on the Netflix Original Atypical and the CBS series The Big Bang Theory, as well as on specific media relating to dementia in contemporary mainstream media.

Textual analysis

Content analysis

Discourse analysis

Focus: Portrayal of autism and dementia in the media (critical paradigm)

Topic 1: Cognitive disabilities in the media

Topic 3: Fighting hunger in the USA Focus: Nonprofit organizations’ use of social media to communicate (critical/ empirical paradigm) Analysis of three highest “liked” Facebook posts from three highest budget hunger nonprofits in the USA, dwelling on use of imagery, tone, feasible call to action, and “shareability.” Review of all social media posts from ten anti-hunger nonprofits for the months of December 2018 and January 2019, coding for aesthetics choices, use of SMART guidelines, and type of persuasive appeal. Review of user comments on all social media posts from 2017 and 2018 on the Facebook and Instagram channels for five national and five local/regional antihunger nonprofits, looking for elements of engagement with the issue or the organization.

Topic 2: Quality healthcare Focus: Provider/patient interaction (interpretive paradigm)

Conversational analysis of ten interactions between staff and patients in the waiting room, dwelling on tone, information provided, and two-way communication. Discussion of forty observed patient/staff interactions in a waiting room, coding for best practices in information conveyance.

Discussion of guides for patient/ provider interactions in medical textbooks, hospital brochures, and training videos, focusing on two hospital chains and two medical schools in the state of Florida.

Four groups of five, one group with people who know someone with autism, and one group people who do not. Discussions about their perceptions of autism after seeing The Curious Incident of the Dog in the Nighttime and how it might alter their understandings or behaviors towards autistic individuals. Identical framework for Forget Me Not.

Although surveys don’t necessarily suit studying the plays themselves, pre- and post-viewing surveys can be an effective way to understand how theater impacts audiences. In this case, asking audiences about their understanding of, experiences with, or personal connections to autism and Alzheimer’s disease could reveal whether the plays expand or change

Focus groups

Surveys

Figure 4.1  Comparing methodologies

Interviews with five spectators of the play The Curious Incident of the Dog in the Nighttime and five spectators of the play Forget Me Not within a week of seeing each play. Participants should include parents and children who saw the film together.

In-depth interviews

Interviews with marketing managers from ten different hunger-focused nonprofit organizations discussing their strategies for social media posts.

Four groups of five, one of social media managers of local nonprofits fighting hunger, one with their social media producers, and one with the target audiences for the organizations. Discussions about intended and actual uses and gratifications from the posts, as well as perceptions of the posts and their strategy overall. A survey here could focus on potential donors, asking them their perceptions of the organization, the issue as a whole, their donation practices, and even specific questions about what they would like to see in a nonprofit website. Ranking various possibilities rather than listing individual ones as important can be very useful here, since for nonprofits there is the risk that respondents

Interviews with ten sets of doctors and patients after their consultation to determine their individual perceptions regarding the interaction’s emotional content, informational content, and next steps. Three groups of five, one of doctors and nurses, one of staff members, and one of patients. Discussions about what makes for a good experience, what are challenges to that experience, and how they think things could be improved.

Providing pre-consultation and post-consultation surveys to patients and doctors is a very effective way to measure whether there is a disconnect in how each feels about the experience, and then those surveys could actually be discussed with the doctors to try to solve any issues that occur. Asking questions about what

In many ways the textual analyses are case studies. However, you could also study cases in which the plays were performed in certain contexts (Forget Me Not) is performed in elder care facilities, for instance. The key to a case study is specificity.

Studying medical archives, media archives, or other sources could provide valuable information about how these medical issues have been understood or portrayed over time. These could be notes, for instance, from production, casting, and direction meetings for shows like The Big Bang Theory, but they could also be from the stage adaptation of The Curious Incident of the Dog in the Night-time.

Case studies

Archival research

viewpoints. A survey done a month or more later as a follow-up, moreover, could reveal longer term impact.

will say that everything is important. Similarly, asking them to respond with specific goals in mind helps to narrow their focus (e.g., “Rank which of these things would be most likely to attract you to volunteer”). The case studies used in this project would focus on specific efforts from one organization, with information about the challenges, goals, and changes made in the process by the managers and other staff.

Looking at the archives of hungerfocused nonprofit organizations could reveal how those organizations (individually or collectively) view their place in society, view their donors and clients, and structure programming. It can also reveal how they have approached communicating about that programming or those stakeholders.

information was conveyed, how easy it was to understand the information, preconceptions about the doctor or hospital, etc. are all important possibilities. Looking at patient-provider interaction in a case study would most likely study a specific hospital, private practice, or hospital ward. The tighter the focus, the better. This is telling the story of that ward, rather than giving more general data. Doctors have changed quite a bit in the public imagination over time. Studying archives of medical materials, from advertising to American Medical Association documents for doctors to educational documents, could provide valuable perspective into the construction of the profession, but also how that profession has viewed patients differently over time.

Experiments would be extremely difficult to do here, because the populations are so vulnerable. It would be easier to conduct an experiment with caregiver professionals in schools or nursing homes, comparing how they treat patients who they have been told have autism or Alzheimer’s disease with those they have been told have another condition, using actors to simulate the experiences.

Experiments

Figure 4.1  Continued

An ethnography of a new production of the plays might be a way to approach this, but you could also conduct an ethnography of autistic children and their families (perhaps in a school) and compare it to another of people with Alzheimer’s disease and their families and professional caregivers.

Ethnography

Studying the lives of clients impacted by hunger is one way to approach the topic through ethnography (living with a client of the organization, for instance), but you could also work alongside the volunteers or organization staff for a set amount of time to understand their culture, their struggles, and their various rewards/triumphs. Experiments for this project could use multiple websites or social media posts for various stakeholders (e.g., different categories of donors, volunteers, clients, etc.) to see their responses. Then those different media could be correlated with other data, like donation activity or volunteer registrations, to determine which types of posts work best with which stakeholder.

In this case ethnography is a good fit and could be used to study residencies by accompanying new doctors throughout their workdays to see how they treat and learn to interact with patients and families.

As with the disability project, studying how patients interact with multiple forms of doctors could reveal a lot about preconceptions and differences connecting due to identity or attitude, and doing the same with doctors could reveal similar biases and inconsistencies in approach.

48  Part I: Get the party started right to answer different questions, connected to the three different research projects in Figure 1.1 (theater and disability, quality healthcare, and nonprofit digital communications).

Creation, portrayal, and representation In this first approach, you’re studying something that has been created to represent the world in some way. Usually this is media: literature, art, music, television, Instagram posts, and so on. It could also be speeches, conversations, and even nonverbal communication like body language. The important thing here is to recognize that something was created to convey a point of some kind, even if it was never meant to be seen or read. It could be a poem written by a musician to himself, or a notebook of jokes written by a comic as part of her process. It could also just as easily be the history of an event as told by a single spectator/ participant or a series of historians over time. There are three major methods in this approach that we’re going to outline for you here, and we’ll go further in depth in later chapters. Each of these relies on what we call a “text,” which isn’t necessarily words (in fact, in advertising, marketing, journalism, and public relations we call written words “copy” to distinguish them from other texts). A “text” could be a poem or novel, sure. It could also be a photograph, a film, or even a person (e.g., an analysis of how a subculture portrays itself through clothing and accessories). Places can also be texts; imagine how your classroom or office is designed and laid out to create a certain perspective. Is there a long boardroom table? If so, who sits at the head, and who at the foot? What’s the table made of, and what’s on the walls? Is your classroom arranged with seats in a circle? Is it a huge lecture hall with a lectern at the bottom and a giant screen for a PowerPoint presentation? How might that arrange the

4: Choosing a method  49

way we think when we enter the room? On what social and cultural backgrounds does that interpretation rely? Close textual analysis (sometimes also called multimodal semiotics) • In-depth reading of one or a few texts, with emphasis on their physical structure (e.g., shooting and editing for film, layout for design, rhetoric for a speech). • Example: While discussing the second season of 13 Reasons Why we thought it was important to look more deeply at the introductory scene Netflix added to the first season after controversy over the show reached a certain level. It alerts viewers to potentially triggering content (that is, there’s evidence to suggest seeing suicide portrayed in the media can increase the likelihood of people who are considering suicide acting on those thoughts). We wanted to see how Netflix created that scene, and what its details might show us about how it could be read. We found that the language used, as well as the dramatic tones and some of the visual choices, actually make the problem worse. They almost challenge viewers, suggesting that to not be able to watch the show would somehow be cowardice or evidence of childishness. Note that we aren’t claiming how it will be read, or what the intent was; we’re just talking about what we see and hear. • Limitations: Textual analysis is a great building block for other methods, and often other methods will feature miniature textual analyses throughout as a way to show examples of elements that they are analyzing through another method. However, at the same time it is limited by the number of texts, and might inherently suffer from ­cherry-picking (choosing a text because it supposedly proves your point). One way to

50  Part I: Get the party started right avoid this is to look for analyses in your literature review, or texts that you can analyze, that contradict your initial findings. Content analysis (qualitative and quantitative) • Brief reading of a large number of the same sorts of texts, usually with the aid of explicit “coding,” in which commonalities and differences are examined across all texts and charted, usually in part for statistical significance. • Example: In the first version of the 13 Reasons Why study, the student began with a content analysis of the first season. The codes she chose were drawn from reports on suicide prevention in the media, what we call “best practices,” drawn from multiple sources in medical and psychiatric journals. She broke those practices into smaller parts, and then coded for the number of times each episode did any of the positive (e.g., “showed someone asking for help”) or negative (e.g., “glamorized death with post-suicide attention”) practices referred to in the literature. She added up the numbers for each code, and was able to determine how much each episode, as well as the entire season, did or did not adhere to best practices for their audience’s safety. In reworking the article, Ted performed the content analysis independently for what we call “inter-rater reliability,” and then he and the student chose certain scenes for mini-textual analyses that could further explain the data with added depth and richness. • Limitations: Content analysis is very useful, especially if you include strong quantitative data combined with strong qualitative elements. However, you can only analyze so many texts, and so it is a challenge to avoid cherry-picking on the one hand, while providing adequate depth to demonstrate nuance on the other. Discourse analysis (often called critical discourse analysis) • Brief reading of a large number of usually different kinds of texts in order to discuss the ideological landscape surrounding

4: Choosing a method  51

a central topic, such as journalism, popular media, educational systems, state institutions, or family systems. • Example: To understand the controversy around 13 Reasons Why, we considered a discourse analysis of reviews and articles about the first season of the show, and perhaps the second season if necessary. In that case we would have compiled dozens if not hundreds of official (from established television critic) reviews, newspaper articles like opinion-­editorials (op-eds), and even reviews from non-official sources like blogs, posts on social media, and posts on aggregate websites like Rotten Tomatoes. This would have allowed us to understand how people were speaking about 13 Reasons, and could have even expanded to discourse about teen suicide (or suicide) more generally. In other words, we would have been able to look at the context and debates that surrounded the show, rather than the show itself. • Limitations: Discourse analysis is an incredible tool, but it often requires a lot of time and space. It’s very difficult to do well either quickly or in a short paper. It is also more qualitative than a content analysis, since you’re engaging themes more than individual codes. However, if you limit your sample size and look for something more specific, discourse analysis can be a very effective tool to study reviews of media, sentiment about an event through social media comments on that event (e.g., through a hashtag search), or other important concepts and events.

Perception/reception In this second approach, you’re not interested in what’s been created. Instead, you’re interested in what people/societies/ eras/governments etc. thought and felt about what was created. In other words, the content of the actual text isn’t as important as the reactions to that text. It’s also important to recognize that

52  Part I: Get the party started right just because you talk to audiences doesn’t mean you’re writing about those audiences. You might, for instance, interview business leaders about their approaches to marketing an environmentally friendly product, but your study is on the approaches themselves. In that case, you’re simply doing what we call “formative research” with interviews, not actually studying the perception of the business leaders themselves. This is a crucial distinction, because it has some pretty big ethical implications. In formative research, you’re trying to get a feel for a topic (just like when you’re doing a literature review) by exploring that topic. That could be watching a TV show, visiting a country, or talking to people. But that’s not necessarily going to be your method.

IRL: ETHICS ARE IMPORTANT When you’re making people (or animals, in some cases) your primary focus of analysis, you might be subjecting them to something that could be upsetting or dangerous, or even simply exploiting them for the purposes of your research without telling them you’re doing it. Imagine, for instance, going on a bunch of dates with someone just for the purposes of writing about dating (yes, that’s a How to Lose a Guy in 10 Days reference). That might work if you’re writing a column for a magazine, but it’s not ethical scholarship unless you’ve told them what you’re doing, how, and why. Most people probably wouldn’t go on that date with you if they realized you were studying them, right? It’s important to remember that your research isn’t as important as the people (or animals) you’re studying.

4: Choosing a method  53

For a study like this one, you’d need them to sign an agreement acknowledging they consent to be studied, and in most cases you would need to send the study through what we call Institutional Review Board (IRB) approval. The IRB (usually a collection of professors or researchers) would then look at what you’re planning to do and with whom, how you’re making sure they’re safe and informed, how you’re compensating them, etc. before allowing you to do the project (see Chapter 7’s online resources for more on this process). This might seem like common sense, but it isn’t. Many studies done even in recent memory have inadvertently traumatized their participants (e.g., the Zimbardo prison studies at Stanford) and even led to their deaths (e.g., the Tuskegee syphilis study). In other cases, millions of dollars have been made from a study and the participants or their families received nothing in return (e.g., the use of Henrietta Lacks’s cancer cells). Whether you’re working in a classroom or a billion dollar company, make sure you do all your research ethically. It’s not that hard, and it actually makes your study stronger.

In-depth interviews • Structured, semi-structured, or unstructured, interviews often conducted with fewer participants but in great depth, usually from 1–2 hours in length; in-depth interviews (IDIs) try to get participants to tell stories and explore concepts in detail, often with follow-up questions or examples. • Example: So say we didn’t really care about what 13 Reasons Why was doing, but instead we wanted to see how

54  Part I: Get the party started right people were reacting to it, or to issues around it. This would be a good reason to do in-depth interviews. We would choose a small sample of people (perhaps ten college students would be a good example in this case), and if we wanted to get more detailed we could even ask students to participate that have seen the show, hadn’t seen the show, or who have even suffered from depression. We would absolutely need IRB approval, especially for the depression group, because it would then be a vulnerable population that would also be potentially at risk through this research process. In any case we would want to alert the campus wellness center about our project, and have a brochure or other materials, as well as contact information for counselors, which we could give to all participants. We should offer people that we interview compensation for their time, either in money or some other form (we like the idea of contributing to a related nonprofit organization like a suicide prevention hotline). Then we would create an interview guide of a series of open-ended questions that would ask participants for stories regarding how the show and other media have impacted them in the past, how they dealt with it, what they would have liked to see or what they were disturbed by seeing, and so forth. We’d audio record these interviews (with permission and a signed release, of course), and then transcribe them into a word processor (and as you’ll see with Chapter 10, programs like NVivo are very useful for this as well). You can use free programs like Speechlogger or paid programs like Trint or Descript for transcriptions to speed up the process. Then we’d read the transcripts and pull out significant concepts, like we did with our literature review: we’d group themes across participants and write them out in detail. You’re trying to bring out a sense of what they’re thinking and feeling, and to do justice to their points of view.

4: Choosing a method  55

• Limitations: Interviews are great, but the sample size is small and you have to be careful to restrict your sample’s range because of that. In some cases, like if you’re interviewing professionals (CEOs, law enforcement, doctors, etc.) this might be the best way to go because of difficulty accessing them and the importance of individual perspectives. But in general it means that while you will probably find some interesting information, its applicability is limited. Interviews are often a great way to both start and end a project. You start with them in order to get a sense of what’s at stake, but you often end with them once you have questions developed through other methods in order to expand and add depth to your data. Focus groups • Small groups of individuals that are targeted to gain specific information about a topic; usually random groups or groups that are intended to get a range of opinions, focus groups are widely used in industry studies of media, like whether a new TV show will be popular, or in social sciences to gain perspectives on things like body image, and so forth; focus groups are great because you can use a semi-structured interview guide but encourage conversation among individuals, which you record and then analyze; however, they can also result in “groupthink” or silenced voices. • Example: Focus groups are often used in media analysis, including by industry professionals who are testing a new show/film/advertisement or deciding on whether to renew/continue/change what they’re already doing. For 13 Reasons Why, we could do focus groups with counselors/ psychologists/psychiatrists about the benefits and problems they can see with the show for their clients, or we could do focus groups with students like in the interview example (both again with IRB approval). In this case, we want

56  Part I: Get the party started right stories but we also want participants to communicate with each other. Discussion is where the richness of this method comes from, and you as the interviewer are actually just a discussion moderator. Usually you’d do focus groups with two or three sets of people that are either all similar (to see if there are differences within them) or different (professionals vs. parents vs. students, for instance) to see if the audience differences translate to interpretation differences. In this case, we might discuss Netflix/television use in general, then dramas, then controversy in dramas. Then we could watch part of or a full episode of 13 Reasons Why before going back into discussion about responses. Participants are paid as with interviews, and the discussion is again transcribed and analyzed through the same process. • Limitations: Focus groups are great for discussion, but they can be bad for data about anything but discussion. That’s because people will often respond with what we call a “response bias,” perhaps regarding what they think the others want to hear or don’t want to hear (often also called a “social desirability bias”). They might make claims or censor themselves in a way that tells us a lot about “groupthink,” but not what they actually believe or even why they answered without complete honesty. Even the person that is interviewing or mediating can produce this effect through what are called “demand characteristics.” In the “Bradley Effect,” for instance, researchers found that in the United States people might claim to vote for a racially non-White candidate, but they might still actually vote for the White candidate. Surveys • A questionnaire of some sort given to a sample of individuals to gain their perspective on a topic; samples are usually large, but depth of reaction is often not as intricate as focus groups or interviews; common scales are Likert scales,

4: Choosing a method  57

and samples range from “convenience” to “snowball” to “random.” • Example: If we wanted to see how a large group of people responded to or thought about 13 Reasons Why or a related issue (e.g., portrayals of suicide, bullying, or sexual assault on television), a survey might be a good method. We would still have to choose a sample, and this could (like focus groups) be a set of groups that have differences or a larger number of participants in a more homogenous group. We’d create questions that ask specific things about the show, perhaps in response to a clip we’ve embedded in the survey, and questions about whether someone has watched and heard about the show, what their perception of including such controversial topics is, and whether or not media companies have an ethical responsibility to their viewers. We would then gather the data and break it down into usable conclusions. That might be percentages of the group who felt a certain way, average responses for different demographics, correlations between two different points, or many other possible analyses. • Limitations: Surveys can be a struggle because it’s sometimes difficult to ask questions that get at to what you want to know, and even harder to do so in a short enough survey that participants don’t get tired and just start filling items out without thinking. We’ll get into this more in later chapters, but while surveys can tell us what the general trends are, they aren’t always the best way of determining the complexities and nuances of how people think. Checking a box among a list of options can limit people’s true thoughts and feelings. Sampling is also difficult, and attracting enough participants to actually answer your questions can be a real struggle. Even when you’re finished and the data is analyzed, moreover, you may want to follow up with other methods to gain more depth and perspective. After

58  Part I: Get the party started right all, maybe you’re only looking at incomplete data because you’re only asking the questions you want to know, and not allowing for respondents to give you contradictory information (e.g., “loaded questions,” “leading questions,” etc.). One other limitation is that people often only respond to surveys if they’re (1) forced to do so, (2) incentivized, or (3) feel passionate about the topic. When you analyze responses from people who feel compelled to give them, you may notice some false polarization (large amount of positive/negative). This is why it’s so much more common to see strong reviews for products than moderate reviews; if something works the way it’s supposed to, or if we have adequate service, we rarely feel that it’s anything special. If it breaks or is bad (or changes your life for the better) you’ll be more likely to write a review. So instead of looking for 5- or 1-star reviews on Amazon, we should probably focus on what the 2-star and 3-star reviews say. Those will provide the nuance or richness of information that will actually tell you if the product is any good or not.

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: Think before you method A word of caution on choosing methods in general: each discipline in the social sciences and humanities has what I like to call “pet methods.” For one thing, as I’m sure

4: Choosing a method  59

you’ve noticed throughout this text, there are certain methods that just work better depending on the discipline, and even depending on the topic you’re researching within that discipline. There are also some major philosophical divides even within disciplines, so even if it’s, say, your fifth class in a particular major, it’s a good idea to work with your professor or a colleague on selecting the most appropriate method for your assignment. Surveys in general seem to create the most amount of controversy. Why might that be? Well, as we’ll discuss in Chapter 8, they can suffer from the same issues as your literature review. Selecting the correct number of participants, the type of participants, and even how you get your message out to potential participants can affect the outcome. Also, surveys are cursed by notoriously low response rates, making matters even more difficult. We’re not saying don’t use a survey, but definitely understand the pitfalls of any method ahead of time. Your librarian can help with these issues, including helping you to design your study if you’re stuck.

Observational analysis There are many ways to approach observation, and we won’t go into most of these here because they’re often different from discipline to discipline. Basically, all of these methods require you to observe something in the real world and then draw conclusions from it. This happens in the methods we have already discussed, but the emphasis here is on the observations and the structure in which the text is observed (the time, place, situation, etc.). Some observational methods have particularly large

60  Part I: Get the party started right ethics requirements, so we’ve highlighted those below in ethics alerts. We’ll look at some more later, but for now we’re just going to highlight a few that you might encounter. Case studies (often used in psychology and business) • Study of a specific instance, often focusing on a business or a person, that might explain or be paradigmatic of a broader trend or issue. • Example: With 13 Reasons Why, a case study approach in this sense could follow one person’s response to the show (e.g., if we were writing it from the perspective of a psychologist). It could also follow the business case for the show, discussing it as a Netflix Original and how/why it became so successful. Similarly, it could follow the public relations challenges faced by the company during the controversy over the show, and how Netflix did or did not follow best practices for public relations in their responses. • Limitations: Like textual analysis, case studies are limited by number. The point here is to show why this particular example is interesting and what we can learn from it, not to make general points about, say, depression or media purchasing or public relations. These are great for cautionary tales or success stories, but we should always be wary of applying one case study to another situation and assuming it will follow the same process or pattern. Archival (often used in history, political science, and metasynthesis methods) • Similar to a discourse analysis, archival research uses an institution’s records about an event, text, process, etc. to determine how it was created or discussed. This kind of research is also used in metasynthesis, which brings together a bunch of scholarly sources to examine their results all together, almost like a qualitative content analysis of scholarship.

4: Choosing a method  61

• Example: One of the most interesting things about any television show is how it was created, from beginning to end. Looking at an archive of materials around 13 Reasons Why could reveal decisions from casting directors (and maybe original casting videos), script changes (e.g., for the third season actress Anne Winters suggested they include abortion in some way), and reactions to controversy, popularity, etc. This gives us insight into what the various people that constructed the television show were seeing, listening to, and talking about during the process of its planning and creation, as well as, of course, the decisions to highlight it on Netflix and to renew it for subsequent seasons. • Limitations: Archival research depends on having an archive, obviously, which doesn’t always happen. It also depends on the researcher recognizing that any archive is going to be incomplete, which will skew her understanding of the topic.

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: Using archives The word “archive” can be used to mean a variety of things, unfortunately, but in this case we mean a physical place that’s usually located in a library or museum with a name like “Special Collections and Archives.” In this repository, the archivist(s) collect and organize collections of history as well as of (often) famous people like artists, musicians, and celebrities. Occasionally, archives will also

62  Part I: Get the party started right

collect and store items that center around a theme. These items might be letters, maps, and other documents, but they could also be artwork, clothing, or recordings. The Archives of Social Change, which was created at the W. E. B. Du Bois Library at the University of Massachusetts, Amherst, is dedicated to preserving materials related to social change in the United States. Archives generally have abbreviated hours and often are by appointment only, so you’ll want to make sure you allow yourself enough time to get in and handle the materials well in advance of your deadline. For more on archives and archival research, see my note in Chapter 6.

Ethnography (often used in anthropology and business) • Investigation of the discourses, events, rituals, and communications within a community or a (set of) individual(s) within that community; although auto-ethnography is absolutely valid as a method, it must be chosen with great care and attention to the method as appropriate for the sort of impact and evidence necessary for the project. • Example: An ethnography of 13 Reasons Why could focus on the filmmaking set and discussions happening as the show is being filmed, or even the editing room and the discussions/decisions made there. It could also be in a high school where some students are watching the show, observing the ways it does or does not enter into their lives and daily communication. In the first cases, we would speak to movie executives to get approval, and then get consent from everyone observed in the study. We would then spend a significant amount of time (meaning, as much time as

4: Choosing a method  63

they spend working) observing the creation of the show, taking “field notes” (miniature textual analyses of discussions, places, etc.), and extracting themes in the same way we might in other methods. • Limitations: Ethnography is a fascinating method, but it is always limited at least in part by the perspective/perception of the researcher, as well as social desirability bias. One of the main issues is the rise of attribution bias or attribution errors, which happens when we assign positive attributes to people and situations to which we can relate to ourselves and our experience, and negative attributes to those that we can’t easily relate to or who are dissimilar. Worse, we tend to assign those positive or negative attributes to their identities or cultures, rather than to the individuals or the situation in which something occurred. Prejudices like the negative construction of “Asian drivers” or “women drivers” draw from such biases and are not only incorrect but can also do tremendous damage.

ETHICS ALERT! In ethnography, you’re observing people in their “natural habitat” and drawing conclusions about them from it. This has led to a lot of problems over the years, from researchers observing simply to justify their preexisting beliefs, to incomplete claims, to the exploitation of vulnerable groups. It’s essential in ethnography, even auto-­ ethnography (the study of the self), to get consent from all people involved, and inform them of what you’re doing and why (and what they might get out of it, how they can

64  Part I: Get the party started right

limit or control the data, etc.). Imagine you’re studying a group and you end up revealing some information about, say, infidelity, identity, or illness that could lead to social stigma or other kinds of harm. This could destroy lives, so you have to be very careful and always remember that (1) you need IRB approval, and (2) you are in a position of power as a researcher, and it’s your responsibility to be responsible and ethical. This is also for yourself, and of course the people in your life that might inevitably become a part of any self-study. The challenge is that when you alert the subjects, they will act differently and thus lose “authenticity,” so you have to find a way around that.

Experiments (often used in psychology, communication, and business) • Within a controlled environment, manipulating variables in one group in order to determine changes.

ETHICS ALERT! Experiments can be very revealing. They can show what we see and don’t see, what we listen to, and what we just hear. They can show us our fears, our biases, our hopes, and how we respond to authority. They can also, however, result in severe trauma. Children in the famous 1970 “blue eyes brown eyes” study by Jane Elliott, for instance, in some cases still feel that trauma fifty years later. Participants in

4: Choosing a method  65

Philip Zimbardo’s 1971 study on prisons at Stanford University are similarly impacted. The list goes on, from Milgram to Tuskegee. Experiments require careful planning, careful consideration of the impact of the study, and careful interactions with participants before, during, and after the experiment. There are so many potential issues that we can’t address them all here. Our example, however, will show you one possible problem and outcome.

• Example: To end with a scary example, we could conduct an experiment in which we took two groups of teenagers that were at risk for suicide, separated them, and exposed one of those groups to 13 Reasons Why. We would measure how differently the control group (that wasn’t exposed) and the experimental or “variable” group responded, perhaps measured by pre-viewing and post-viewing surveys and perhaps by conversations with a therapist. The risk here is of course that being exposed to the show could aggravate their tendencies to self-harm, and even result in someone’s suicide. Experiments can be dangerous things, so tread lightly, do your literature review exhaustively, and work with professionals in all cases. And always, always, always work with your IRB. • Limitations: Experiments are fascinating but, like every research method, they also have their limitations. Easily the biggest critique of experiments is also one of its strengths: the need to control. In other words, in order to see if what you’re studying is causing a particular change in a group of people, you need to focus on that variable and minimize ever other possible influence on the group. So, if you want to see how a film or TV show might affect people’s

66  Part I: Get the party started right emotions, you would need to make sure that you remove as many other possible variables as well (e.g., other people, room ambience, screen size, volume, etc.). The best way to make sure that happens is by bringing them into a laboratory to control and standardize every other possible influence. However, in controlling and standardizing, you have just now created an artificial environment that may or may not reflect reality. Ironically, the very same laboratory room you are using to control other factors actually is creating its own effects on your participants. So, yes, experiments can be a great way of isolating and focusing on how specific variables may change the outcome of a group, but anytime you remove human beings from their natural contexts (e.g., living rooms, classrooms, etc.) you run the risk of telling an “artificial” story.

Although this often wouldn’t be the best method for this question, you could analyze suggestions made by employees through an anonymous email survey, or responses made during exit interviews when employees leave the company. Code for tone, similar concepts that might be positives and negatives, and forcefulness of feeling. As sites like Glassdoor start to provide extensive, anonymous reviews of companies, it’s increasingly important to monitor them to determine what employees are saying when they work for you and when they leave. This can head off some serious problems, showing you issues with ethical

Analyze all of the complaints and other communication that comes through email and on social media, like comments on posts, tweets about your products or customer service, and hashtag use. Code for tone, issue, and positivity/negativity, as well as whether it was resolved and/or responded to by your staff. One of the most useful things you can research is how people are speaking about your company and its competitors. You could look at social media accounts of “influencers,” or just look at what people are saying when they hashtag you or comment on your own social media. You could

Compare your competitors’ social media presence to your own, coding for types of messages, media, demographics, even tone (in addition to likes, shares, etc.). Target at least the past year.

Because discourse analysis is about perception and social construction, it doesn’t really help much when figuring out what your competition is doing. However, it could certainly reveal opportunities and where you and your competitors are succeeding or failing. For instance, when you

Content analysis

Discourse analysis

Figure 4.2  IRL: How to do what your boss asks

Do a textual analysis of a break-room or an office space, connected to best practices in the industry for productive workers. For instance, Clif Bar & Company has an office filled with athletic equipment, and an in-house massage therapist and gym.

While it’s possible to look at customer blogs, videos, etc. if you have an issue (e.g., with United Airlines), usually you wouldn’t choose this method because it’s about perception, not portrayal.

Look at how your competitors are advertising their products and if there are missing demographics or situations (e.g., selling SUVs to “soccer moms” instead of to dads).

Request: Are our employees happy? What can we do to make them more productive?

Request: How do our customers feel about us, and are we doing well or badly?

Textual analysis

Request: Find out what the competition is doing and where there’s opportunity for us to grow.

If you want the most rich, detailed information on why people are choosing your competition, in-depth interviews can really become immersed into the perspective of the customer. Rich descriptions and unique stories can help paint a clear picture of why customers/clients are choosing your competition.

To help flush out some of the major stories why people are choosing your competitor, you can conduct focus groups to hear from several people at once. Focus group research can help remind and confirm the attitudes associated with competitor allegiance.

In-depth interviews

Focus groups

learn that people are interested in socially good products, or companies with progressive politics, you can tailor your message and your products or services to match.

violations, bullying, inefficiency, and other things that you should fix.

Similarly to the customer interviews, you might select employees from different departments or levels in the organization, and administer your IDI. While you can expect richness of data with this method, you most likely will spend time looking for common themes that emerge from the qualitative data. A focus group to determine employee satisfaction might be useful, however, in order to protect anonymity and avoid groupthink, you’d probably want it administered by an outside facilitator. For this particular question, a focus group might not be the best method. Surveys and especially IDIs might be better in terms of providing rich feedback in a setting that avoids bias and makes your participants feel as though they can speak candidly.

also look at reviews on Amazon, or newspaper coverage related to what you do.

By selecting a few representative customers (for example by location, revenue amount, or company size), and conducting IDIs with a set list of questions. Be careful to avoid leading questions of course, and if possible get a third-party to administer the IDI.

Focus groups can be extremely helpful in determining how customers feel about your company. This method can also be helpful when attempting to make claims about which particular groups like or do not like your company, based upon the makeup of the focus groups you build. Focus groups can sometimes suffer from “group think,” so be careful of this pitfall.

An ethnography of a group that you want to sell your product to is a great way to get good information. You can easily figure out what people buy, when, and how much they spend, but ethnography helps you truly understand the many reasons why.

Experiments are a great way to confirm whether specific variables are causing a specific change in a group of people. In examining a competitor’s business strategies, you could determine if a specific strategy is influencing customer acquisition or sales.

Ethnography

Experiments

Figure 4.2  Continued

You want to find out what are the most common reasons people are flocking to your competitors, conducting surveys can help you find trends and patterns. It can also identify how high/low customer’s attitudes rank when they purchase your competitors products

Surveys

Surveys provide employees with the anonymity they need to speak freely about the company, but you won’t get the same amount of qualitative responses as from an IDI or focus group.

It’s important sometimes to get a feeling for what work is like on the ground. Are people overworked? Do they feel undervalued? What do they worry about, and what might make their lives better and easier? Those answers are rarely as simple as “more money,” because you still need to know why. Again, this would be far more involved than some of the other methods, but you could develop interventions to determine employee productivity and satisfaction and compare the overall scores across the conditions.

Surveys can be useful in getting a wide array of responses fairly quickly, but you might limited on how much information you get. It’s nice to have a summary of overall attitudes, but you may not get the whole story.

Embedding yourself into campus tours, dealership test drives, and other situations can help you gain a close look at how people interact in those strange situations. What do families talk about when they’re buying a car or choosing a school, and what are the things you didn’t expect? The challenge and benefit of experiment is that the variables you are measuring need to be exceedingly clear. When assessing attitude/affinity often experiment design researchers will assess hormone levels and/or brain activity to determine how much someone likes/dislikes a company// person.

Methodology overviews

PART

II

CHAPTER

5

Close textual and thematic analysis

OVERVIEW AND OBJECTIVES Chapter 5 is the first of our methods chapters, and we’ll take you through how to analyze individual and small sets of texts. This can range from media, to people’s clothing, to places, to situations (e.g., how different cultures traditionally handle it when people meet for the first time). We will discuss how “multimodal semiotics” helps you frame this approach, but we’ll also address how textual and thematic analyses can be used to determine relationships, power issues, and social norms.

Textual analysis has many names, probably because it comes from different places and perspectives. This method was in many ways developed by historians and scholars in the visual arts, music, and literature. It looks at the elements of a text to determine what is occurring within it, without in most cases making claims to audience interpretation or “author.” Before going any

5: Close textual and thematic analysis  73

further, it’s worth looking at those in a little more depth. While scholars often refer to the “author” (e.g., a painter, poet, or director of a film) as the creator, it is also usually implicit that what is actually being read is the text, not the intent behind the text even if we use phrases like “Shakespeare is creating a tragic flaw in Hamlet.” After all, who knows what the author was actually doing, no matter what they say they were doing? In reality, it doesn’t matter and to make this kind of claim is what we call an “authorial fallacy.” Similarly, while we often say things like “which makes us identify and empathize with the character,” that’s not really making a claim to an audience interpretation, but instead what you as a scholar are suggesting is a likely audience interpretation. Otherwise, you’re committing an “audience fallacy,” since you’re analyzing one thing but claiming things about another.

IRL: DON’T ASSUME. THINK AND ASK In one famous marketing failure from 2017, Pepsi released an advertisement featuring Kendall Jenner in which she joins a crowd for what looks like a street protest full of beautiful, multi-ethnic people. They are confronted by a line of police officers, and Jenner prances to the front of the protest so that she can give an officer a Pepsi. He drinks it and is happy, and so then everyone plays music and they’re all friends. This was a not-so-subtle reference to a Black Lives Matter demonstration against police brutality, in which a young activist named Ieshia Evans calmly stood her ground against charging police in militarized riot gear. It was also a reference to the famous “Flower

74  Part II: Methodology overviews

Power” photograph from 1967 taken by Bernie Boston, in which protesters against the Vietnam War placed flowers in the muzzles of rifles pointed at them by a squad of soldiers. In either case, the advertisement was not only in general bad taste, but was also extremely offensive to much of the demographic Pepsi sought to target (culturally open, diversity-focused millennials). Anyone on staff should have been able to immediately see the problems with this ad, and should have spoken up. Even if they didn’t see it, they should have asked others for their interpretations. This is a great example of how a failure to use even basic methods of critical research can lead to a public relations crisis, in this case a crisis that cost the company millions of dollars.

Textual analysis is often called “close textual analysis,” and some scholars call it “multimodal semiotics,” which means the study of signs in a text from a variety of perspectives. Understanding what “signs” are in this case helps explain textual analysis as a whole. Think about it this way. Let’s say someone draws a hexagon for you. OK, great. You have a shape. Congratulations. However, when Ted does this in his classes, invariably most of the class interprets the hexagon as a stop sign. If it was colored red, obviously the connection would be even stronger. However, nothing about a hexagon says “stop,” which means that our perceptions and beliefs are so embedded within our culture that a simple shape immediately means (or “signifies”) something profound. And what’s so profound about a stop sign? Well, it means “stop,” of course. But really only if you’re in a car, right? And only if you’re driving that car, right? And does

5: Close textual and thematic analysis  75

it only mean “stop?” Who’s telling you that, and why? What are the consequences if you don’t stop? A stop sign is actually saying “stop . . . or else.” That last bit is important, because while a stop sign is signifying “stop,” it is also signifying a system of ethics in which cars should be careful to avoid people, other cars, and other objects, as well as signifying a system of power in which if you don’t stop, you might get a ticket from a police officer. Either way, it’s going to be a bad day. So that’s textual analysis. You look at as many details as you can to get an understanding of all the different levels that build the sign, or what we call “signification.” We’ll take you through some examples throughout this chapter that will make this clearer. Details in a textual analysis come from many different places. In literature they might be the way words work together (meter, rhyme, assonance, etc.), in film they might be lighting, placement in a scene (mise en scène), and color, and both might examine setting, narrative, and dozens of other elements. It’s helpful to categorize textual analysis into some general concepts: formal/aesthetic (how it sounds, looks, etc.), allusions (references to other texts), narrative (the story), setting (place and mise en scène), etc. All of this depends on what exactly you’re analyzing. So let’s start with a couple of possibilities. 1. Art: One of the most famous examples for textual analysis is a 1929 painting about textual analysis, René Magritte’s The Treachery of Images. It’s a painting of a pipe floating with no context on a tan background. Under the pipe in French it says “Ceci n’est pas une pipe,” or “This is not a pipe.” So . . . ok. What’s going on here? There are only two signifiers, really: the pipe, which is painted fairly realistically, and the words (sometimes called “the copy”). Of course, this is a pipe. But it’s also a painting of a pipe, and when it’s shown in class it’s a computer projection of that painting of that pipe. But the pipe is also a representation of other things,

76  Part II: Methodology overviews right? Perhaps masculinity. Perhaps home. Perhaps upper class “distinction.” Perhaps hobbits. But what Magritte is saying here (or rather, what the painting or text is saying) is that we need to think deeply about any signifier, because it rarely only means one thing. 2. People: Maridath, Josh, and Ted are all shameless peoplewatchers. There’s nothing quite like trying to figure out what someone is thinking, what their background is, and who they really are just by examining them from a distance. But what do we look at? Nonverbal communication like gestures, clothing (“artifacts”), eye contact, hair, personal hygiene, the volume of their voice? Verbals, like the words they use, the language they’re speaking? Setting, like the place and whether they “fit” in that place? Context, like the social situation we see them in and the people/events/objects around them? The answer is all of those things and more. A crazy bachelorette party would look completely normal at a club in Las Vegas. Take the same party and put it in a breakfast truck stop diner in a small rural town at 7 am and it looks insane. Deep textual analysis looks at all of these things, because the richer your data the less likely you are to make mistakes due to bias or your own situation and mindset. 3. Place: Places are fascinating to study, even boring ones like a classroom. How do students know to sit in their regular seats, rather than at the front of the room? Are the chairs in tiers like a lecture hall, or small tables where students look at each other, or in one big circle where everyone sits down? What color are the walls? Are there interesting paintings on the walls, or are they blank or dull? Is the furniture made of old, expensive hardwood, or mass-produced industrial plastic? What does all of this say about a university’s perspective on education? On the students? Do they trust students to pay attention? Do they trust students to have respect for the faculty member? Do they value student voices as much

5: Close textual and thematic analysis  77

as the professor’s voice? A textual analysis doesn’t just look at what’s visible; it also looks at what is invisible, whether that is something that is erased (like women and non-White authors from many literary canons) or something that is simply not present but could be (like a round table in a boardroom or dining room instead of a long rectangle). 4. Television: One of Ted’s favorite examples of textual analysis is the opening credit sequence to a show that he grew up with: The Cosby Show. It’s an incredibly important television show for the history of US and global media, since it was unheard of for a middle-class Black family to appear as the center of a situational comedy (sitcom) full of mainly positive, affirming humor. It was a wild success both in the USA and internationally, with audiences of all backgrounds. We can see how that happened in part through the opening credits. It is a simple set of still images with copy placed in front of them that tells the audience who the actors are, starting of course with Bill Cosby (whose character, despite the name of the show, is actually named Cliff Huxtable). The narrative takes place through a series of successive images, in which a family gets out of a van and plays baseball, basketball, and other games in the park. This is the most “American” of American images: a park, a happy family, a van, and baseball. At the end we discover that these images are all memorialized in photographs placed in a photo album. Signifiers of race and class other than “middle class” (sweaters, jeans, and other simple clothing) are removed entirely, as is any conflict. In other words, it is the perfect image of a post-racial society in which race, class, and other categories no longer seem to matter. 5. Advertising: Flaws with textual analysis often start with limited analyses, letting the ideology of the research guide the data rather than allowing the data to speak for itself. In one infamous ad, the fashion brand Dolce & Gabbana released a

78  Part II: Methodology overviews controversial image that many claimed depicted gang rape. Indeed it could be read that way. One man with no shirt on is holding a woman down on the ground, while three other men stand watching (two of which are at least partially shirtless). That sounds bad. But there are other signifiers as well. They are standing in a place that doesn’t really exist, with a sky full of clouds that reaches to the horizon, water that is somehow in the scene as if they’re floating on an ocean, and huge slabs of steel or concrete that make up the landscape. Even the men aren’t all on the same plane; one of them actually seems to have his legs end impossibly halfway in the ground. So this is a fantasy. But whose fantasy? The men are mostly clad in tight denim (except for one in a frilly purple dress shirt), and their skin is shaved hairless and covered in shiny oil. They are all both thin and extremely muscular. The woman is on the ground, but her pelvis thrusts upwards, her face does not look at all unhappy, she is wearing what seems to be a satin version of a one-piece swimsuit with strappy four-inch high heels, and her makeup and hair are perfect. So is this the fantasy of men who are coded as gay (and gay men are certainly one of the target demographics for this company)? Is it the fantasy of a woman coded as wealthy (the other target demographic)? Or is this the fantasy of one of these men? If it’s his, is the man interested in the woman, or the other men? Is it of abuse, and if so why are her hands not clenched, her arms not straining, and her pelvis lifted up to meet him? Is she “going along with it,” or is she an active subject rather than an object? As with any analysis, incomplete analysis is bad research and leads to bad conclusions. Whatever the interpretation is, make sure it deals with all the signifiers available. 6. Situations: Textual analysis can be very useful as a way to understand and compare various interpersonal interactions. For instance, you could look at how staff at the

5: Close textual and thematic analysis  79

DMV treat people that come to the counter (like the sloth scene in Zootopia), or how people communicate when they match on Tinder. One of our favorite examples is introductions. If you compare how people introduce themselves to each other in different cultures, you can tell a lot about what that culture values and how. For instance, in Japanese you say your place of employment first, then your family name, and finally your personal name. In the United States, however, people will usually just give their first name, only in certain business situations also adding your last name and your company. Unlike in Japan, you as an individual are the priority, not your family or the company.

IRL: WHEN WOULD YOU USE THIS? Textual analysis is one of the most common tools in business across all industries, whether that is understanding what another team is saying in negotiation through body language versus verbal signifiers, analyzing a sales brochure to make sure it accurately and effectively portrays a product or brand, or analyzing a movie trailer to make sure it does the film justice and doesn’t needlessly harm or offend an audience. Similarly, listing a house on the real estate market requires knowing how to portray that house so that the intended market will be able to imagine themselves living there, and asking for a higher salary depends on correct interpretations of your boss’ state of mind, goals, and feelings about your performance. Each of these things draws in large part from textual analysis.

80  Part II: Methodology overviews Ethnography uses many of the same techniques as textual analysis, although it includes other elements like being fully embedded in a community and getting consent from that community. This method has grown to emphasize what the anthropologist Clifford Geertz called “thick description” of an event, society, people, and so on. There are many fantastic examples of powerful ethnography, some of communities like street gangs and some of scholars themselves (e.g., experiences with their own alcoholism, in what we call “auto-ethnography”). One of the most famous examples is a 2001 book from Barbara Ehrenreich called Nickel and Dimed, which describes how she lived as a minimum wage worker to examine issues of systemic inequality and access. We will discuss the method a bit more in Chapter 7, but it is important to understand that while some of the techniques are similar, ethnography is centered on people and social interaction, whereas textual analysis emphasizes the text itself.

For further reading Geertz, C. (1973). The interpretation of cultures: Selected essays. New York, NY: Basic Books. Ehrenreich, B. (2001). Nickel and dimed: On (not) getting by in America (1st ed.). New York, NY: Metropolitan Books.

CHAPTER

6

Content and discourse analysis

OVERVIEW AND OBJECTIVES This chapter builds on Chapter 5 to show you what happens when you want to analyze a much larger number of texts. Studying a large number of texts would take you forever in textual analysis, but content analysis gives you a systematic, repeatable, and more easily correlated set of data to draw from. It’s one thing to watch one episode of a TV show or one “State of the Union” speech. It’s another to watch twenty of the same type of text and see how they’re similar or different from one another. Content analysis is great because it is both a qualitative (type) and quantitative (number) method, and you can choose how far you want it to be on either side depending on your goals. Here we’ll give you some suggestions on how to choose a sample and then “code” your data, making it easier on yourself.

82  Part II: Methodology overviews OK, let’s take a step back. Textual analysis is awesome, but it’s also limited. When you look at that many signifiers, how do you know what’s important? How do you know that what you’re reading into the text has broader significance beyond the text (e.g., to other advertisements by the same company) or connects to audiences the same way (e.g., what is it from the viral video that made it viral, and was that a good thing or a bad thing)? Those kinds of questions really need a higher sample size (N), and while this will sacrifice some of the depth, mini-textual analyses of specific examples can often be a useful way to incorporate depth into other, broader methods like content analysis or discourse analysis.

Content analysis The first of these methods, content analysis, tends to lie somewhere in between a qualitative (studying the qualities of texts) and quantitative (counting something in or across texts) method. In this case, you look at a series of texts that are related in some way. You want them to be as similar as possible, so you might look at a series of advertisements in the June issues of five fashion magazines, or all speeches from a certain president, or a large set of poems about love, or the Corporate Social Responsibility reports from leaders of a certain industry. Similarity is important because you’re going to ask a series of questions about each text that you can answer quickly and easily. We call those questions “codes,” and we’ll use that term a lot. Codes could be “how many people appear in this advertisement?” or “Is the person in the image making eye contact with the viewer?” or “What percentage of the report is on technology?” When you’re coding, it helps to have a spreadsheet where you can put the codes as columns and the texts as rows, and then quickly go through each text. Hopefully you can have most codes answered as, for

6: Content and discourse analysis  83

instance, yes/no, a number, or a percentage. You’re trying to reduce ambiguity. However, in some cases you are looking more for qualitative data, in which case you might code for “emphasis” rather than presence or absence. In that case, you might rate things on a scale from 0 (not present) to 3 (highly emphasized) or in categories like ones used to describe social media posts for a nonprofit organization (event promotion, story of community impact, request for donations, announcement of volunteer opportunity, etc.). As we’ll mention later in Chapter 7, it’s sometimes helpful to look for elements that occur multiple times within a text, that occur strongly, that occur across texts, or that seem surprising in comparison to other texts. This might seem overwhelming, but it’s actually a very efficient way to go about analyzing a lot of data. It’s systematic so it can be reproduced, you can easily add to your sample of texts, and you can find similarities and differences (as well as connected or correlated data) quickly. One thing to remember is that you’re most likely going to want to add mini-textual analyses to your data. While numbers are all well and good, it’s important in most cases to give some examples that show your readers what you’re talking about. For instance, doing a content analysis of all speeches given by Barack Obama before he was elected and then while he was president is going to give you some cool numbers (e.g., the percentage of words and/ or time that he devotes to reaching out to people that disagree, the percentage that he devotes to reaching out to those that agree, and the percentage that he devotes to bringing the two sides together). However, it won’t give you the richness that a couple of examples might offer, especially from famous speeches that might extend the analysis (e.g., “Yes We Can” or “A More Perfect Union”). The important thing here is to draw a series of codes in the same way that you might make categories for a textual analysis. For a nonprofit’s social media, for instance, you might

84  Part II: Methodology overviews have categories like “aesthetics,” “storytelling,” and “call to action” in addition to numerical data like reach (“likes”) and engagement (“shares” and comments). Those categories would then be broken down into many subcategories. Storytelling for instance could have a code for voice (Who seems to be speaking? The organization, a volunteer, or a staff member?), a code for content (What’s the post about?), the intended audience (donors, volunteers, clients), the call to action (donations, an event announcement, a thank you to a donor or volunteer), and other similar concepts. These are all answered either as yes/no questions with a code for each, or as a list of questions where you can answer from a set of predefined answers that you build by doing a quick run-through of your sample. Similarly, aesthetics could focus on whether there is media present and what kind (video, live video, still image), how much copy is present (words and/or characters), how professional the media is (1 for not professional, 2 for not bad, 3 for very professional), and whether the colors suit the brand’s standards. The list is potentially endless. However, you need to start with your research questions. Once you know what you want to find out, you can determine what codes make the most sense for you to look for. You’ll probably end up with a fairly long list (probably closer to thirty than ten!), but the longer the list, the easier the process will be later on. We suggest doing this in a spreadsheet program like Microsoft Excel where you can easily compile or average data, add or remove rows, highlight cells, and make tables and graphs so you can visualize your results. One of the best ways to approach a content analysis is in groups, where you and at least one other person can come up with the list of codes (the “coding scheme”) together, and then you can do the analysis (“rate” the texts) independently. Before you’ve done the ratings you can also code a few texts together in what we call a “norming session” so you’re sure everyone agrees on what each code means and how to interpret the categories.

6: Content and discourse analysis  85

After you’ve rated them, you can then calculate what we call “inter-rater reliability,” which basically means you’re figuring out how close you were in the analysis. Using Excel or SPSS, you simply tabulate each of the coders scores in adjacent columns and run a Cohen’s Kappa (two raters) or Fleiss’ Kappa (3+) analysis to determine the consensus (or lack thereof) between the coders. If you score 82%–100% reliability (k =.90) your coders are probably sharing a brain. Strong reliability is within 64%– 83% (k =.80 –.90). Much less than that, and you will want to consider having another norming session. This type of analysis allows you to see how subjective your material is and whether or not the best way to describe it is through quantitative data, or if it only allows you to see general trends that you’ll then have to spend more time explaining in deeper, qualitative descriptions. See our online resources for some step-by-step guides on how to do this in Excel.

IRL: BREAKING DOWN THE COMPETITION One of the best uses for a content analysis is to compare your organization or client to similar examples (including competitors). You could even do this for yourself when you’re looking for a job, or on cars before you buy one. This is an incredibly valuable skill, and it is a quality that distinguishes good managers from bad ones. In his work as a marketing consultant, for instance, Ted often compares what a company offers to what competitors offer in terms of specific customers. He draws codes from best practices in marketing, but he also draws codes from what

86  Part II: Methodology overviews

those customers want (to see, hear, etc.) drawn from other studies and from interviews or focus groups he has conducted. As a health management consultant, Josh compares the various interactions patients have with doctors, nurses, and other staff across not just the one hospital he’s working with, but also other hospitals. He codes for best practices to increase satisfaction and outcomes found in the literature on patient-provider interaction. Maridath works with multiple library databases to determine what they offer for their prices, coding for not just what they have and how they work, but also what the university faculty and students need to access, and how much they overlap with the existing content the library offers. These are all content analyses, even if they aren’t always done in the same systematic way that we have advocated here.

Examples of content analysis codes always depend on your research questions, but a few examples can illustrate how it might work. 1. Advertising: Say you want to compare how gender appears in multiple forms of advertisements, not just the Dolce & Gabbana example we discussed above. In that case you could choose every ad released from Dolce & Gabbana (and perhaps a competitor or two) in the past year or several years, or perhaps every ad in certain magazines or websites within a specific timeframe. The first approach would show you how Dolce & Gabbana and/or its competitors construct their advertisements through the use of gendered or sexed individuals. The second approach would allow you to

6: Content and discourse analysis  87

compare how different outlets construct gender and sexuality, and connect them to their “media kits” or “press kits” that tell you who they’re targeting and how (or simply infer audience from their articles). Codes would be drawn from a few textual analyses showing us what elements might be important, like various ways in which men and/or women might appear. If you’re asking a different question, those codes might be less relevant, but in this case, they’re essential. For instance, if you’re studying the wedding industry, you might want to look at how De Beers constructs diamonds through their famous campaigns starting with “A Diamond is Forever,” and look at how the situation constructs gender through wealth and social class. Different questions, but same method. 2. Television: If you are analyzing The Cosby Show, you might want to look at how sitcoms as a whole construct identity over time. In the first, you’d use the textual analysis we did above to form the relevant codes, expanding as you go. You could choose every sitcom’s opening credit sequence over a set period of time, coding for things like the role images of the city might play in the narrative (something you’ll find a lot in credit sequences), the impact of music (e.g., essential throughout, for instance in Diff’rent Strokes, The Jeffersons, or The Fresh Prince of Bel-Air), and the visual appearance of each character. You could also do a more quantitative analysis looking at numbers of people that appear or are coded as different races, and see how often and when we see integrated or segregated content. In gender studies, for instance, we have what’s called the Bechdel Test, named for comic artist Alison Bechdel. With the Bechdel Test, scholars code films or television shows for one simple criteria: whether two women ever speak together about anything other than a man. This is a search for gender portrayal and equality in the media, and many subsequent scholars

88  Part II: Methodology overviews have expanded this to other criteria to show further levels of depth (e.g., whether the women are named or are sexualized). Scholars have also often used content analysis to look at how children’s programming constructs, silences, or erases history and identity, particularly of girls, non-White ethnicities, and the economically disadvantaged. 3. Social media: Social media is usually produced in large quantities over time, so it’s an ideal candidate for a content analysis. To stick with our fashion theme, one way to use social media in a content analysis is to look at how fashion “influencers” (people that are followed for their style or perspectives on fashion) portray themselves on Instagram. You would then choose perhaps ten individuals, perhaps five celebrities and five non-celebrities, and code their past twenty-five Instagram posts. This is important because then you’re coding a variety of people, but also crucially looking at all posts, not just ones that are about fashion. That would allow you to determine whether something else makes them an influencer, like sexuality, images of food and working out at the gym, or even “authenticity” like photos of pets or casual days at home with their family. Taylor Swift, for instance, made a name for herself in large part on the perception of her authenticity on social media, and that authenticity itself is something to be studied since it encouraged what we call “parasocial relationships” with her fans (that is, the imagined sense that they “know” her). 4. People: One of the ways in which content analysis can be used to study people is through conversation analysis (sometimes called “discourse analysis,” although we use that term in a different way). In this case, you might study how people speak about fashion in a clothing store, and code for differences depending on the people involved (e.g., ethnicity, male and/or female, customer and/or salesperson, and every combination thereof). Do people mention body

6: Content and discourse analysis  89

weight? Sexiness? An event, a romantic partner, a desired “feel?” While this might sound a bit like ethnography, and it is in fact similar, the difference here is how you’re going about looking for information and then processing that information.

SAMPLE CONTENT ANALYSIS CODING SCHEME Focus: Food bank Facebook posts Target of Analysis: Twenty most recent posts Food bank name Date of post Time of post Hashtags (#) Hashtags (list) Length of post (words) Link? (presence/absence) To what? (category: blog post, external news, donate page, volunteer signup page, other) Media? (presence/absence) Image? (presence/absence) Of what? (category: facility, volunteer packing food, volunteer serving food, employee, client, client eating food, client working, child, child eating food, child working, food, boxes of food, other) Tone? (category: funny, cute, sad, inspirational, informational, other) Aesthetic quality (emphasis: 1 = low, 2 = medium, 3 = high)

90  Part II: Methodology overviews

Graphic? (presence/absence) Of what? (category: map, infographic, drawing/animation, event marketing, other) Tone? (category: informational, funny/cute, inspirational, other) Aesthetic quality (emphasis: 1 = low, 2 = medium, 3 = high) Video? (presence/absence) Of what? (category: donation process, sourcing process, client profile, inspiration, program information, impact information, volunteer profile, donor profile, employee profile, other) Tone? (category: sad, happy, inspirational/hopeful, angry, detached, other) Aesthetic quality (emphasis: 1 = low, 2 = medium, 3 = high) Length? (minutes) Clear call to action? (presence/absence) To do what? (volunteer, donate, attend an event, other)

Critical discourse analysis While content analysis is a useful tool, it is not as useful for understanding the broader context surrounding an issue or event. In other words, content analysis is very useful for studying the same type of text with a limited timeframe or sample, but it’s not as good at seeing how the issues themselves are socially determined (e.g., what “fashion” means, what “woman” or “girl” means, what “Black” means, etc.). Usually when we think of those questions we turn to historians, who often search through archives of

6: Content and discourse analysis  91

material compiled by individuals, universities, museums, libraries, and other sources. Historians are concerned not just with what happened, but what people said about what happened and how different people portray or construct the event. Similarly, political scientists, environmental policy scholars, law scholars, and other researchers will often look at specific archives of materials to determine the history of a law, policy, or tradition, and how it has been applied or constructed over time. Often there are huge archives of materials, including letters, drafts of documents, and other related items that can provide crucial insights into the history of a time or place. Archives of city planning maps and development approvals can tell you about the changes in a town and its connections to racial or ethnic identity (e.g., gentrification). Presidential libraries can explain how a certain decision was made, and what advice was given during debates around pivotal events (e.g., the Cuban Missile Crisis). Art libraries can provide sketches and images of studies that reveal the origins and possible other end results for a film, a photograph, or a painting. However, when scholars want to discuss something perhaps more current than can be found in an archive, they often move to “historiography,” or writing about the writing about history. The reason for this is that if you’re looking for something that might have been silenced, it makes sense to feel around the issue rather than just look at the issue itself. If you go back to the “blind men and the elephant” story from Chapter 3, in which five blind men each touch a different part of an elephant and make claims as to what it is (the trunk is a snake, the leg is a tree, etc.), historiographers often look at how we can get a more complete picture of the puzzle, perhaps by talking to each of the men rather than trying to describe “reality” itself. Imagine trying to explain something invisible; you might describe what it does or has done, or how others react to it. That’s what we’re calling “post-structural discourse analysis,” or “critical discourse analysis.” Basically that just means that we’re looking at how a

92  Part II: Methodology overviews social, cultural conversation builds and maintains a concept without necessarily talking about it directly. For instance, changing “global warming” to “climate change” or “friendly fire” to “collateral damage” doesn’t alter what you’re talking about, but at the same time it does alter your perceptions. Changing the word “truck” to the phrase “sport-utility vehicle” to “SUV” also changes things, as does “station wagon” to “crossover” or “used car” to “pre-owned car.” In other words, we’re not talking about reality, we’re talking about how we imagine and construct what we think of as reality. The most famous discourse analysis scholar is the philosopher, historian, and psychologist Michel Foucault. He studied not madness itself, but madness as constructed by civilization, and did the same with prisons and punishment, sexuality, and even history itself. After all, what we define as “crazy” or “criminal” or “a genius” has changed radically over time, and the story of who is crazy (or gets to be crazy) is tied to other issues of class, gender, race, and privilege of all sorts.

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: Using the archive and others’ archives There comes a point where you might need to consult some primary sources for your research. What do we mean by primary sources? For starters, this definition can get quite slippery, and like a lot of other definitions in this book it’s going to vary by discipline. Here, what we mean is original physical (or digital) documents created by members connected to the topic or event you are researching. These

6: Content and discourse analysis  93

archives usually live in your college or university library, and many also live online. For the physical archives, you might need to make an appointment with the librarian (usually called an archivist) to view the collections, usually during specific hours of the day. Because the materials are often rare and fragile there are often rules that must be followed to gain access. So what might you find in the archives? For example, if you were researching a famous painter or musician or other figure, their archive might include items such as correspondence (letters), journals, and items archivists called “ephemera” – which is literally “stuff.” These can include the sketches that then became the paintings, drafts of speeches, even letters to mistresses! Like your secondary research, and your selection of methods, it’s a good idea to consult several archives’ materials on one person or topic to avoid the dreaded cherry-­picking. Your librarian can help put you in touch with your college’s archivist, and can even recommend archives at other museums or universities that you might consult.

Discourse analysis derives from a mixture of fields like anthropology, sociology, political science, and history. The focus is not only that language constructs our reality, but also that the construction is an active and constantly evolving process, a social practice of sorts. That means that it often looks at conflict, and tries to understand what’s going on from the differences that might exist (or the silences that exist) within various forms of social discourse. It also often draws its analysis of these competing forms of reality or history from a large number of texts that come from different types of media, different

94  Part II: Methodology overviews sources, etc. You will most likely code for themes just like you would in a content analysis, but probably not at the same level of detail because you are looking at longer and/or a larger number of texts. The key here is that you’re not studying the texts themselves, you’re studying what they’re saying about something else. You’re still looking for patterns, though, or what discourse analysts often call “interpretive repertoires.” There are a few things you can focus on: outliers, or what have been called “deviant” cases; consistency and coherence with previous work on the topic (e.g., your literature review); understandings of an event or issue (e.g., letters to the editor, movie reviews, advice columns). You can also simply present your data and ask readers to come to their own conclusions. Here are some examples of discourse that might help you get a feel for it: 1. Business: One of the most interesting ways to use a discourse analysis is to look at how the media, employees, analysts, and internet commentators might construct an issue or company. One of the most famous examples in more recent memory is the electric car pioneer Tesla, whose stock is known for being unpredictable and volatile. It is guided by comments from CEO Elon Musk, earnings reports, conjecture from analysts about suppliers and customers, commentary from amateur investors on sites like Seeking Alpha, media coverage of the company or CEO (even his love life or use of recreational drugs), and public conversations with investors. Each source has its own agenda and background, and the concept of “Tesla” is completely different from the company itself. In fact, this is true of almost everything in business, since business is built on future opportunity as much or more than past success. 2. History: Discourse analysis is used quite a bit in critical history studies. For instance, an analysis of textbooks

6: Content and discourse analysis  95

across the nation (or globe) could reveal quite a bit about how various regions treat (or conceal, erase, and misrepresent) the Civil War, slavery, and indigenous populations. Connecting that to records of school board meetings in which boards, parents, and teachers meet, as well as the presence or absence of discussions about this issue on the news and social media, can provide a snapshot of how the culture is actively constructing and silencing certain histories. 3. Arts: Art isn’t just made of the artifacts themselves; in fact, that is only a small part of what we know of as “art.” The well-known, award winning, controversial Broadway musical Hamilton, for instance, could be studied in terms of how news sources, professional and amateur reviewers, and actors spoke about the play. This was particularly interesting as actors came out actively to contest popular understandings of history, noting that their versions in the play weren’t accurate, but that neither were the versions in history books. In other words, Hamilton and the discourse around it was in many ways a struggle over who gets to write history and how. 4. Brand perception: Airlines are an oligopoly of companies that everyone hates. No one likes to fly, but many of us have to put ourselves through the misery of airports, security lines, immigration, and tiny seats just to save some time. They’re all around the same price, with mostly the same service. But perceptions of the airline can make the difference, as United Airlines has found out repeatedly. Internet sensations such as the “United Breaks Guitars” song from Dave Carroll or the infamous video of security personnel beating a passenger as they dragged him off a plane in order to give his seat to an airline staff member, have caused incredible damage to the brand. To fix it, any analyst would need to look

96  Part II: Methodology overviews closely at the news coverage around the events, but also social media responses to the various videos, memes that are being circulated, and ways in which other airlines like Southwest capitalized on the brand’s failure to deal with the crisis effectively. Coding a discourse analysis is similar to the process for coding a content analysis, but it’s rare to get yes/no categories here, and you’re usually not comparing “apples to apples,” but rather different media, times, or channels of communication. For instance, analyzing political speech could look at speeches, YouTube comments, news coverage, and opinion-editorials around a single event. What you’re usually coding for is particular categories of discourse and, perhaps, content like tone and rhetoric. In this example you might look for categories like “approval/disapproval” or “positive/ negative,” but you could also look for references to other events or political figures, inclusion of data, references to other news sources or studies, or rhetorical devices like fear appeals, emphasis on hope, or discussion of collaboration. Numbers here could portray the percentage of comments online that say one thing versus another, but also percentages of speeches and news articles devoted to various issues and perspectives. Discourse analysis is a challenge because it introduces much of the complexity and depth of a textual analysis with the large number of texts in a content analysis (even more, in many cases). However, the key here is that you aren’t actually doing thick description, you’re simply looking at how prevalent or intense certain kinds of discourse are in various media. In other words, you aren’t looking at the true depth of a text or set of texts, you’re looking at how those texts frame an issue. The issue is what is central here, as well as ways of looking at/speaking about or erasing/concealing that issue. That makes it an incredibly powerful research tool.

6: Content and discourse analysis  97

SAMPLE DISCOURSE ANALYSIS CODING SCHEME Focus: Reviews of Will and Grace 1998 original and 2017 reboot Target of Analysis: Twenty newspaper articles for original and twenty for reboot Source Author Date URL Length (words) General response Positive (yes/no) Negative (yes/no) Neutral/mixed (yes/no) Funny? (yes/no) Predictable? (yes/no) Nostalgic? (yes/no) Controversial? (yes/no) Discussion of positive portrayal of homosexuality (yes/no) Emphasis on positive (words) Emphasis on political/social change (yes/no) Discussion of negative portrayal of homosexuality (yes/no) Emphasis on negative (words) Discussion of homosexual content as deviant, weird, etc. (yes/no) Emphasis of “weirdness” etc. (words) Discussion of impact on viewers? (yes/no) Relevant to American culture? (yes/no) Reboot different from original? (yes/no)

CHAPTER

7

Ethnography, interviews, and focus groups

OVERVIEW AND OBJECTIVES Unlike text-based analysis, which is centered on what a text itself is doing, the methods in this chapter are all concerned primarily with human interaction, perception, and reception. Ethnography is a kind of “participant observation” that embeds you within a social group; interviews are deep discussions with a strategic sample of individuals, and focus groups are broader conversations with small groups. In each case, while the data you collect is also analyzed and coded, what you’re looking for is how people interpret their reality, how they tell stories about that reality, and how they create knowledge through actions, rituals, and social systems.

7: Ethnography, interviews, focus groups  99

Let’s refresh. The reason we do research is to capture and report knowledge, or the “truth,” about something. For some, the truth is a single thing that is out there to be discovered, regardless of how an individual perceives it. For example, if a doctor is trying to locate the source of your headache, they could make a decision based on fMRI scans rather than your description. You probably aren’t a doctor, after all, and you might thus mislead them (as Dr. House would say, “people lie”). The interpretivist researcher, however, sees truth as subjective depending on who is observing and who is being observed. “Truth” then can be best understood as someone’s perception, often conveyed through conversation. So, after visiting the medical doctor you may also visit a psychologist, who diagnoses that the source of the headache as stress at work. You get some therapy to avoid brain surgery. Both healthcare providers were able to locate a “truth” about your headache, but in the second case that truth comes from the messiness and complexity of life. Interpretive research comes from the belief that really interesting “truths” are as much in our heads as in the world outside, and the research goals focus on how we perceive and receive reality. Three of the most common ways of doing this are through ethnography, IDIs, and focus groups.

Ethnography One of the most interesting, but also most problematic, methods of research you can conduct is ethnography – from the Greek ethnos (people) and grapho (I write), and meaning “writing about people.” In this method, you usually embed yourself within a group and take notes (what we call “field work”) about the people in that group. This is a kind of textual analysis, but in this case you’re studying the details of people and their interactions from within the social system itself. The process is fairly simple. You choose a target group to observe, you join

100  Part II: Methodology overviews that group in some way, and you observe them, taking “field notes” along the way that include detailed information about events, verbal and nonverbal communication, and other elements. These notes can then be coded in a similar way to textual analysis or discourse analysis, which we discuss in depth in Chapter 10. It’s important to remember that what you’re coding for and how you’re coding for it can change, depending on your questions, background and discipline, and what your literature review has taught you. There are also three primary ways to conduct ethnography: undercover ethnography, non-participant observation, and participant observation. Undercover ethnography This method was often used to study groups from within as a completely embedded participant. That means that researchers would claim to be a part of that group (like an undercover police officer) in order to observe participants as organically as possible. There are many reasons to do this. For instance, what if you want to study prison culture? Gangs? Human trafficking or organ trafficking? All of those have been the subjects of undercover ethnography. However, at the same time we have to be very careful here. Because the participants can’t give consent if you’re undercover, you can end up exploiting and misrepresenting them. That means that undercover ethnography is rarely if ever ethical as research. Leave it to the police or journalists, who have a different job to do. Undercover ethnography is also misleading, because it’s very difficult to be completely organic as a part of a group if you’re not actually a part of that group. That means that if you are separated a bit through the act of observation and covert note-taking, you might introduce elements of bias and expectation that can lead to changes in the behavior of the people around you. For instance, if you’re studying gang violence and are undercover, should you be violent? And if you feel you need to be violent for “authenticity,”

7: Ethnography, interviews, focus groups  101

does your act of violence influence others to be more or differently violent? The answer is usually yes, it does. In other words, authenticity here is actually impossible, which means that being undercover is often self-defeating. It can never be truly organic and might skew your results even more towards what you think a group should be doing, rather than what they would do if you weren’t there. Non-participant observation In this method, you’ve received IRB approval and informed consent from all participants (see the online resources for some guides on how to do that successfully). However, you are attempting as much as possible to not participate in the group’s interactions. This could happen through video recording, but there are challenges in terms of the depth and detail that can be captured. Of course, being there and taking notes might also reduce depth, since you won’t catch everything and might be immersed in some detail or emotional moment (e.g., a riveting story from a recovering alcoholic in an Alcoholics Anonymous meeting) and miss something important (e.g., that half of the room isn’t paying attention to that meeting, or that someone in the back is shaking from alcohol withdrawal). The biggest challenge here, however, is that simply the act of observing something can change that thing (according to particle physics, it always does). Certainly having an observer present during an AA meeting could change the dynamic of that meeting, especially if people are communicating trauma. Video recording the interactions might even make the problem worse. Since researchers are also given a kind of cultural power and authority, this impact could be massive. In other words, it’s almost impossible to have non-participant observation; the best you can get in most cases is a limited form of participant observation. Imagine yourself as a child playing with friends. Even if your parents aren’t in the room, you know they’re there and often act accordingly. If

102  Part II: Methodology overviews they’re in the room it might be worse, and certainly if you’re being recorded it might be worse still. This is the same principle. Participant observation Participant observation basically means that you’ve gotten IRB approval and informed consent, and you consider yourself an active participant in the group you are observing. That means that you’ll have to observe yourself as well, and of course in some cases like auto-ethnography you’re focusing primarily on yourself. However, in most cases you’re simply acknowledging the biases, changes, and limitations to observing interactions in any organic or “authentic” way. That means that your work will need to be deep enough to acknowledge what is going on in what you’re observing, but also engage how you might be impacting the events and interactions. It’s a great way to try to understand group interactions, but challenging to do both ethically and in a way that provides truly useful data. Combining ethnography with other methods can often be a useful way to add elements that might be missed, especially if it happens before and after your observations. Surveys are great here, but so are the other two methods we’ll discuss in this chapter: in-depth interviews and focus groups.

In-depth interviews In-depth interviewing is an acquired skill. Many people think they’re going to be good at it but often aren’t. Your first time conducting an interview may not go as planned. Perhaps your participant isn’t talking as much as you thought they were going to. Maybe you’re going through your questions too fast. Maybe you’re not pausing enough, or not long enough. Then suddenly the interview is over, and you’re not sure you have anything significant. Uh oh. Relax. Good interviewer skills take time and practice to develop. Let’s discuss some important principles.

7: Ethnography, interviews, focus groups  103

Bias You have bias. Admit it, then try your best to avoid it. While you can conduct an experiment behind a one-way mirror and make every effort to remove your presence from the study, as long as you’re studying it, the bias is still there. Qualitative interviewers make no apologies about their involvement and participation in co-creating and reporting on the truth of people’s experiences. Remember, you have just developed an extensive literature review, a research design, and a set of research questions about your project. At this point, you are knowledgeable and have a good idea about what people think/ feel about your topic. That is a good thing, but it can also be a problem, since you might end up accidentally guiding the research towards what you’re expecting to find. There are some ways to reduce this effect though, and creating a good interview guide is a good start.

IRL: REMOVING BIAS FROM A JOB INTERVIEW The literature shows us that people might not get jobs even if they’re qualified, due to a variety of reasons ranging from the color of their skin to the assumed ethnicity of their name to assumptions about their socioeconomic class. Bias could also be a result of the club or group they were a part of in university, or the fact that they did or didn’t play a certain sport. With this many variables, it’s challenging to remove bias. Removing names from resumes is one possibility, but how do you do that when applicants have to list other information that includes their name, or when part

104  Part II: Methodology overviews

of what you’re evaluating is the creativity and design of their resume? One of the best ways to eliminate bias is to locate it and discuss it openly as an interviewer before looking at any materials. Think about what might cause you to be biased, why, and whether it actually should disqualify someone for the job. That alone can make a huge impact. In one fascinating example, orchestra administrators were confused as to why there were so few women that made it to higher positions. Some people said that men were just better, others that they were more passionate musicians, but most were skeptical of both explanations. So they did an experiment in which they would start concealing the musicians performing for juries (the groups of judges that choose whether you get in or not) behind a screen. What happened? A lot more women and people of color made it through. Then they realized that juries could still assume a gender by what they heard when the person came out on stage, since high heels made a distinctive sound. They had everyone wear slippers, and again more women made it through. These juries didn’t think of themselves as sexist; they just had preconceptions in their heads about what a “genius” looked like, even though what they were supposed to be measuring was how they sounded. This experiment was an attempt to create a “double blind” interview, where neither the participant (the musician) nor the judge (interviewer) knew who the other was, what they looked like, and so forth. This is one reason the peer-review process for scholars is so important, as well. It shouldn’t matter who you are; it should only matter how good your ideas, methods, data, and analyses are.

7: Ethnography, interviews, focus groups  105

Interview guides One of the first actions you can take to help curb your own bias is to construct an interview guide, which is a prepared script with a list of questions that you’ll use as you do the interview. There’s a balancing act here between very scripted and very unscripted, though. Do I write out every question and deliver it verbatim? Is it okay to ask questions that aren’t on my script? Do I need to stick with the same order each time? The best interviews will often feel like a natural conversation, and at the same time, it’s important to understand that structure is necessary to compare responses and prevent leading questions. In a “non-structured interview,” there is no interview guide at all. Often corporate executives who have been conducting employment interviews for several years will claim that they need no prepared list of questions. They allow the interview to flow like an organic conversation and make decisions based on intuition. There are a few downsides to this interview approach: 1. It becomes difficult to compare responses across your population if you ask wildly different questions between participants. 2. Interviewers who don’t use guides will subconsciously ask more “leading questions,” which means that they unintentionally guide participants’ responses, often because they mean to establish “common ground.” There are distinct differences between an unscripted conversation and a guided research interview. Conversations build relationships, and both individuals are seeking common ground and areas of agreement. Non-structured or low-structured interview guides can actually build a false sense of agreement, however. The interviewer may unconsciously hint at their personal attitudes about a topic, which often results in influencing (and perhaps changing) a participant’s response. Imagine you’re interviewing for an internship, and your future boss says, “So,

106  Part II: Methodology overviews it says here on your resume that you lived in Boston for a while. I’m a huge Red Sox fan – are you?” Well, if you weren’t before the question was asked, our bet is that you are now (and will go home and burn your Yankees jersey). On the other side of the continuum is the “highly-structured” interview guide. In this case, interview questions are written and delivered verbatim. We call this format “Stick to the Script,” and it’s not unusual when there are many interviewers speaking with many candidates (say, for entrance into college). You state them exactly as they are written, in the exact order, with no deviation. Comparing multiple responses is much easier to do in this case, but there are some problems: 1. It stifles any opportunity to make the interview feel like a natural conversation, which can increase the sense of nervousness and even the interviewee’s desire to answer in a way they feel is “right” rather than “true.” 2. It doesn’t allow for improvisation or for following unexpected pathways, which can remove important elements that could impact your study or decision. 3. It doesn’t allow for a great deal of depth, which comes from more natural interaction and storytelling. We suggest that you aim for the middle ground, a “semi-­ structured” interview guide. This approach is both natural and structured. Participants should feel as though the conversation you are having is unique. However, as a researcher you also want to make sure you are rigorous and neutral in collecting information, and so you have a pre-scripted list of questions and ­follow-up questions that cover each topic. The conversation can flow naturally and jump from question to question, and if some responses seem both pertinent and interesting, you can ask participants to “explain a bit more” or to “give an example.” Yes, it takes a bit more skill to navigate a semi-structured than a highly

7: Ethnography, interviews, focus groups  107

structured interview guide. Anyone can simply “Stick to the Script.” But with time and practice, you become more comfortable with the balance of a natural yet structured conversation. Questions Like interview guides in general, the questions you ask also range from open to closed. On the one hand, highly open questions give the participant complete freedom to respond in any direction they would like. Questions such as, “Can you tell me about yourself?” and “What does it all mean?” can be useful sometimes, especially if the participant starts going in a direction that you didn’t anticipate; however, questions that are too open can often lead your participant to go off on a tangent and take up way too much of the time you’ve set for the interview. Remember that at some point you’ve got to get to the questions you need answered! On the other hand, highly closed questions are great for short interviews, and like highly structured interview guides they make responses easier to compare. Unfortunately this also means that participants will often just try to answer your question, which reduces the chances of unexpected directions and responses, or even the richness of detail that interviews can provide. They can also make the interview feel cold and distant, reducing the likelihood that your participants will tell the truth or disclose information that might be more controversial, personal, or uncomfortable. For instance, a closed question might ask something like “did that make you happy?” or “did you have a close relationship with your parents?” but it might not encourage the participant to explain why they felt a certain way, how they felt if they didn’t feel a certain way, or examples of situations that shaped their feelings. We suggest that the key for a useful in-depth interview is to focus on having enough structure to facilitate getting to the point, but that you also focus on having your participants tell stories and go deeper (see Figure 7.1). The deeper you get into their examples, the more data they show you rather than tell

108  Part II: Methodology overviews you, the better. Good examples of moderately open questions might be “tell me about any challenges you experienced during a recent trip you took to the doctor,” or “explain how viewing this advertisement makes you think and feel.” In other words, if you’re conducting a qualitative in-depth interview, we advise you stick to the semi-structured guide. Ground each of your questions in a specific topic/issue to give your participant some direction, and save the more closed, limited response questions for surveys (which we’ll get to in the next chapter).

Highly open questions

Moderately open questions

Highly closed questions

Strengths *Gives participant completely freedom to respond in any way.

Your happy place *Participant has both direction and freedom in their response *Allows for participants’ response comparison *Scripted questions prevent interviewer bias

Strengths *Saves time * Easier to compare participant responses

Weaknesses *Takes more time *Prone to leading questions and response bias

Example What is health?

Example What was your experience like at your last visit to the doctor?

Figure 7.1  Types of questions

Weaknesses *Limits the freedom of the participants’ response and prone to selection bias Example Were you satisfied or dissatisfied with your last physician visit?

7: Ethnography, interviews, focus groups  109

Primary and secondary questions Think of primary questions as the main points of an interview guide outline. They introduce each distinct topic you hope to cover in your interview. The secondary questions are usually f­ollow-up questions that will bring out more in-depth information. A good semi-structured interview will have both primary and secondary questions laid out. While the number varies depending on your research questions and the time you have for the interview, we suggest you prepare around ten to fifteen primary questions, with one to two secondary questions each. This isn’t a random number; more than fifteen questions will most likely lead to “interview fatigue,” which means that a tired participant will often give you short responses because they are tired and want the interview to end (or worse, they’ll answer in the way they think you want them to answer to avoid follow-up questions). In many ways, the interview questions are a conversational guide aimed at addressing your research questions. So, you might start by rewriting your research questions in ordinary language your participants can relate to. For example, you could change “What behaviors lead to high levels of communication competence in healthcare?” to “Tell me about your doctor’s communication style.” You can also draw from your literature review, especially the theories that are guiding your research. For instance, in the literature on disability, scholars often discuss the ability for media to create empathy with viewers. Therefore you could change a research question like “How do audiences experience empathy with disabled characters” to “How did you feel about Christopher? What made you relate to him, and what was difficult to relate to?” This may take a few drafts, but make a long list of primary questions that covers each topic of your literature review and your research questions, then narrow it down to ten to fifteen questions. Keep these questions moderately open, and then develop and write out one to two secondary questions to gain more detailed information.

110  Part II: Methodology overviews Primary question:

Secondary questions:

What communication challenges, if any, have you experienced with your physician?

1. If you can, please describe a story that best illustrates this. (Ask for story.) 2. What about this experience made it so challenging? (Probe for more info.) 3. Is there anything else about the challenges that you can think of that you have not yet told me? (Clearinghouse probe.)

Figure 7.2  Primary and secondary questions

How do you “follow up” on information you don’t have yet? This actually isn’t that hard, especially if you have a good literature review. Ultimately, secondary questions clarify and/or gain more information. If you have an introspective, talkative participant, you may rarely need to use them, but always have them on your interview guide and ready to go just in case you get someone that is shy, somewhat hostile, or just doesn’t know what to say because they’re uncomfortable with the situation. Regardless of how your participant responds, you can always ask for a “story or experience” that grounds responses in a reallife event. You can also probe for more information by asking what specifically made it so challenging, or you can always throw in a “clearinghouse probe” at any point in the interview (especially at the end) which simply gives the participant a second to think about “anything else” they might want to say. Keep in mind that some of these issues might not be at the forefront

7: Ethnography, interviews, focus groups  111

of their minds. When a participant spends time thinking and talking about life experiences, new examples and stories might surface from prior questions. The clearinghouse probe is a great way of collecting those accounts. There are a few problems you can run into when designing your questions, and we want to run you through those so you can avoid them. 1. The “awkward pause”: This is when you ask your participant a moderately open question then, quickly, follow up with a bipolar question. For example, you might ask your participant to explain how they felt about their physician’s communication at their last visit. After you ask the question, though, your participant pauses and you can sense that they are struggling to find the right words. It’s ok to let them think about this for a while. Don’t be afraid of seemingly long pauses (and yes, the pauses seem much longer to you than they actually are). This means you have asked a good question that requires some thought. 2. The “bipolar trap”: This is when you jump into a pause (or don’t even wait for one) before attempting to “save” your participant by interjecting: “So, would you say you were pleased with the communication with your doctor?” Your participant looks relieved and responds “Yeah . . . pretty much.” What could have been an in-depth response about the nuances and complexities of your participant’s communication experience has now been reduced to a simple yes/ no response. 3. Leading questions: Are all of your questions neutral? Have you removed all subtle hints about how you may want participants to respond (and have you thought enough about what you want or are biased to want)? Remember that an interviewer has a great deal of power over the interviewee; often without realizing it, social desirability bias can make

112  Part II: Methodology overviews an interviewee give the answer they think you want instead of how they truly feel. The fact is that people who are being questioned often want to please the interviewer and give the “correct” answer. As you read through your questions, ask yourself: Do any of my questions seem leading in one direction or the other? In asking the question this way, does it seem like I want them to respond in favor or in direct opposition? An example might be something like “It must be difficult communicating with a physician who focuses on their computer during your consultation, right?” You can rephrase that to remove the leading elements: “What challenges, if any at all, have you experienced in communicating with your physician?” is much better. It’s not just problems you should watch out for. You should also remember that there are a few things that can make things a lot easier on you, saving you time or even saving an interview from being unusable. Here are a few: 1. Rehearse: Once you have written your interview guide, be certain that you test it by reading it out loud to a friend that doesn’t know anything about the topic. The questions may sound great and make perfect sense to you (because you wrote them), but the phrasing, word choice, and intent may be confusing to other people, especially those who haven’t done the research and so might not know what certain terms mean. Always practice the entire interview before you begin interviewing research participants. If there is any confusion, rephrase the questions. Make it as simple and clear as you can. 2. Recruiting: We’ll spend some more time discussing sampling methods in Chapter 8, but here we want to talk briefly about how you might recruit participants for your in-depth interviews. In most cases you’re probably looking

7: Ethnography, interviews, focus groups  113

for a specific population with specific criteria. For example, if I want to find out how families of those who suffer from alcoholism cope with their situation, I would probably look for an Al-Anon group. Selecting people like this, from a specific population, is called “non-probability sampling,” since it’s not a random sample of everyone. If you’re looking at an Al-Anon that is local, or at students from your university, you further restrict your sample to “convenience sampling” (a sample that’s easy to get). “Snowball sampling” is useful when it’s difficult to find established groups. With this method, you find one or two participants and ask them to connect you with others in their community that might also be good participants, and then you ask those to connect you, and so forth. 3. Willingness: In order to get permission to interview people in the group, you will need to draft both a recruitment announcement and a consent form. You’ll need to run these through your school’s IRB, and then send the announcement to a director within the group who will either post or send it to members. Remember, you need and want willing participants. Once you have a list of willing participants, find a neutral setting that won’t alter responses and get their written consent before starting. 4. Recording the interview: Don’t assume someone is willing to be recorded without asking them first, because it’s unethical and might also be illegal. You need, in writing, a signed document from your participant that they are absolutely okay with you asking personal questions about the issues related to your research, recording those responses (via notes, audio, and/or video), and perhaps using their accounts as excerpts in your project. When they responded to your announcement, after all, they may not have realized what the process entails, and the might be uncomfortable with it (especially audio and video recording). Before you

114  Part II: Methodology overviews begin the interview, you need to let them know all of this through a prepared written statement about your project. Let them know that: 1. You will be recording the interview, transcribing it, and removing all personal names and places, even though you may use excerpts in your project. 2. Some of the questions may be personal in nature – and at any time they can choose not to respond. 3. At any time, you will stop the recording if they ask you to. 4. At any time, they can leave with no hard feelings. 5. If the interview stirs up any unanticipated emotions, you have contact information for counselors if necessary. Put all of these statements in your written consent form, read it aloud to them, and then ask them to sign it if they are still willing to participate. If they say consent, then you can begin recording. Small digital recorders are fairly easy to find, and your phone can do this too (just make sure you turn it to “silent”). Please record through two devices, and test them first to make sure the audio is usable. Trust us. Let them know when you begin recording, place the recorder(s) on the table, and begin your questions. 5. Take field notes: Focus on the questions, creating a natural dialogue, and record any notes you think might be necessary. We like to write down the setting, time of day, length of the interview, and any nonverbal elements that you think might be significant. For instance, if there was an emotional response to one of the questions or a noticeable pause, write it down. You might find that over half of your participants became emotional over one of the questions, and if so it’s a good thing you noted it from the start. In the end, you’re not sure what will and won’t have a significant impact on your findings. To be safe, write down more than you think you might need.

7: Ethnography, interviews, focus groups  115

6. Transcribe: The audio recorder frees you from the burden of writing down every response. Later you’ll transcribe the interview verbatim (yep, every single word) so there’s no need to worry about that now. Transcribing in-depth interviews might be the most painful task of any research method. You can outsource this job to one of the transcription services we’ve included in the online resources to this chapter, which range from professional human transcribers to programs for your desktop that let you slow down the speed of the file to digital text-to-speech programs. Depending on how fast you type, transcribing an interview takes around two to three times as long as the actual time of the interview (that means twenty one-hour interviews could take you easily between forty and sixty hours!). Transcribing your interviews will make you more familiar with their data and nuances, though, so we suggest you transcribe at least one personally. Since you have a semi-structured list of questions, it’s important that you maintain a natural conversation feel with your body language. Your posture should be engaged and inviting. You want your participant to feel comfortable opening up to you, while at the same time, not showing bias by rewarding “good responses” with overly affirmative reactions. After the interview is over, thank your participant and let them know that their story is valuable. Remind your participant that the interview has added great value to your research as it will add necessary insight about the topics you are studying. In-depth interviews are a great way to get to know the details of your project and your participants, and it can reveal a ton of information you never expected or even thought to look for. What they don’t do, however, is show you how a group of people might interact with your project. Interviews are therefore a little artificial and more focused on past, remembered experiences

116  Part II: Methodology overviews rather than experiences as they happen or are negotiated within a social setting. If you want to get more social information, you’ll need to conduct focus groups.

Focus groups The process for creating focus groups is fairly similar to that for creating interviews, but sometimes it gets a little more challenging because you’re trying to work around many people’s schedules. We like to use online signups like WeJoinIn or Doodle for these, depending on who we’re including in the focus group. Both services allow you to make meetings easily at a time that’s convenient for everyone. In many cases, you’ll estimate around four to five people per focus group, although many use much larger numbers. The reason you want to keep it small is to reduce issues of groupthink (when people start to fall in line with each other in order to preserve a sense of community) and power (when one person dominates a conversation). On the other hand, sometimes those interactions give you valuable insight as to how discourse plays out, and you can get a feeling for people’s responses in post-focus group surveys or interviews if you’re worried about people that didn’t speak as much. The goal, however, is to have a conversation. Interviews are often about deep individual storytelling. In focus groups you’re trying to learn more about how participants relate to one another, and how they can learn from and bounce off of each other, than you are about individual responses. Focus groups use interview guides in a similar way to interviews, but you’re more a moderator than an interviewer in this case. Often researchers will bring in media such as magazines for participants to flip through or show short videos and ask for responses/discussion. This allows you to see how the various groups relate to specific content, and how they figure out

7: Ethnography, interviews, focus groups  117

differences in interpretation. Because of the challenges with equal time or groupthink, if you have specific questions you need answered across the board, pre-focus group and post-focus group surveys or short interviews are often very useful.

Final thoughts In each of the methods in this chapter, remember you’re performing a kind of “participant observation.” Basically, in this case it’s important because you should recognize that your presence itself will modify what is said and how. If you’re speaking about racial tensions and you’re considered a different race or ethnicity than your participants, will that change things? Probably. If you’re discussing body image and you look like a model, or if you dress completely out of fashion, will that impact the results? Probably. If you are interviewing people who are of a lower social or economic class than you, or are younger or of another disenfranchised group, will your interpretations of them be biased? Again, probably yes.

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: Navigating the IRB The IRB is set up at institutions to protect the rights of human subjects (participants in your surveys, focus groups, and IDIs) and to promote standards of professional research on campus. When you’ve selected methods and you know you’re going to involve human participants,

118  Part II: Methodology overviews

it’s a good idea to get IRB approval. Not only are your librarians familiar with this part of the process, but they are often even members of their institution’s IRB. They can also help with finding quality human participants for your research. Because they are engaged in their own research and have an idea of other research that is happening on campus, they might even be able to put you in touch with groups on campus or in your wider community that could serve as resources or participants.

The key is to make sure that you’re attempting to remove bias as much as possible, as well as to get both institutional consent and approval through an IRB and individual consent and approval from your participants (or their legal guardians if they’re minors or otherwise unable to consent due, say, to cognitive disability). It’s also important to remember that you aren’t looking for certainty, or an objective way of knowing, here. You’re looking for the depth and complexity of perception and reception (that is, interaction and belief). Researchers more interested in objective certainty, who are looking for data that can be more easily quantified, correlated, and/or predicted across large samples often turn, instead, to surveys.

CHAPTER

8

Surveys

OVERVIEW AND OBJECTIVES This chapter discusses one of the most broadly used research methods: surveys. In it we will discuss how to do a survey “right,” meaning by thinking intentionally about your research question, your sample and population, the questions you need to ask, and the scales participants will use to respond. We will also give you an overview of some mistakes that are often made in survey design, and how to fix them. This chapter should help you avoid common pitfalls and produce something that is valid, useful, and doesn’t drive you crazy when you try to analyze the data.

Surveys are one of the most widely used (and misused) methods that you see “in the real world.” In some ways that’s because they’re so common, and seemingly so easy. We’ve all responded to a survey at some point, and we’ve all given one. After all, “How many of you like pizza?” is a survey. Simply put, surveys

120  Part II: Methodology overviews are just a tool for collecting consistent data from a large sample. Unfortunately, like anything that’s overused, many surveys are poorly designed and don’t gather meaningful data. Part of the reason is that there are just so many types of surveys and so many types of questions you can ask. They can range from small questionnaires about your customer experience to more openended inquiries about why you quit your job. Before developing a survey, you should ask yourself: What do I want to know? Who do I want to ask? How do I want to send the survey out, and will that impact responses? In the end, you’ll need to consider your audience, the format, the questions, and the overall layout to get the most out of your results.

Sampling: who do I want to ask? If you’re conducting surveys, your goal is to be able to identify patterns across a large group of people. You’re trying to figure out what the majority of your participants think about something. Advertisers, for instance, love to rattle off how much of the population favors one thing over another. But “four out of five dentists agree our product is the best” doesn’t tell you much in itself. Did they survey every dentist in the world? Only five dentists? How many dentists would you need to survey in order to be reasonably sure about your conclusion? This is where “sampling” comes in (see Figure 8.1). It would be impossible and unnecessary to survey every single person in your “population” (e.g., all dentists). There are two main ways to cut down your population: probability and non-probability sampling. Probability sampling Probability sampling is a method for randomly selecting participants, usually using computer programs (e.g., Microsoft Excel or SPSS) that number and randomize an entire population.

8: Surveys  121

Figure 8.1  Populations vs. samples

Random sampling is when you try to get a representative sample of the population (like in the national census or some big public opinion polls). It’s intense and a lot of work, especially if you use systematic sampling, which selects one out of every three people (or every fifty, or any number depending on how many you need). This method is a great way to curb selection bias, and it will help ensure that your sample is an accurate representation of your population. But is it always possible? Let’s say you wanted to study the perceptions of college students in the United States. Say you had a computer-generated list of every college student in the country. It would be impossible to reach out to each of them to take your survey. Probability sampling works great in theory, but it isn’t how most people conduct surveys. Non-probability sampling If you have a specific population in mind, you might need to turn to another method that doesn’t rely on the general population. There are a few ways you can do this. In convenience sampling, you select people close by or who you know to take your survey (e.g., team members, classmates,

122  Part II: Methodology overviews friends and family, or your social media friends). Purposive sampling happens when the researcher strategically decides which people will give meaningful responses. If you want to know how people feel about your company or product, you might want to sample only current customers, or people who patronized your store in the past year, for instance. Lastly, snowball sampling is beneficial when you are dealing with a population that is difficult to locate. For example, if you’re studying how alcoholism affects families, you might only be able to locate a few willing participants. After they finish the survey, you might ask them to help connect you to others who fit the population criteria and are willing to participate, and then ask those people for similar connections until your sample grows big enough.

IRL: MAKING YOUR SURVEY HAPPEN Getting your sample is one of the most daunting parts of doing a survey. There are some automated services that can help with this, though, and our online resources guide you through a few. Mechanical Turk, an Amazon company, can get you thousands of participants very cheaply. A little more expensive but usually better quality service comes from the company Prolific Academia, which also recruits and filters participants. If you’ve got lots of money, you can also use a consulting firm like Gallup, Deloitte, JD Power and Associates, and other organizations that can often help you speak to customers and other specific groups. When you’re ready to actually conduct the survey,

8: Surveys  123

two of the biggest online services in both academic and business settings are Qualtrics and SurveyMonkey. They provide a user friendly online format to create, send out, and collect survey data quickly. Many advantages of online surveys include cost savings, data collection efficiency, and geographical dispersion, as well as maintaining complete anonymity to help reduce social desirability bias.

Survey design and question types Unfortunately, just because you know who you want to talk to and how doesn’t mean you actually know what you need to ask. Before you write your questions, it helps to look back at your research questions and/or hypotheses. For instance, universities want to know how effective faculty are at teaching (RQ1), and how “good” the courses are (RQ2). One of the ways colleges find this out is by surveying students about their perceptions of their experiences with the faculty member and the class in the Course & Instructor Evaluation that you take for every class at the end of the semester. The reason colleges want to know this information is because they want to measure faculty performance and be sure that students believe they’re learning. But does perception actually mean objective performance, especially compared across disciplines, classes, and faculty? Probably not. Still, student perception is a useful way to understand part of the story, and it helps to ask several types of questions. You can use numbered scales (e.g., a Likert-type scale), a set of possible responses (e.g., a categorical scale), yes/no questions, and open-ended questions that ask for examples of your experiences. A good survey will have an assortment of all different kinds of questions aimed at trying to see what a group of people’s

124  Part II: Methodology overviews attitudes are about something. Here are a few examples, each with its own strengths and limitations.

1. Demographic questions Even though each individual response should be anonymous, your audience will probably want to know some demographic specifics about your sample. Common demographic questions include age, race/ethnicity, and gender, but you may also want to know a few other things. Studies on businesses might want to know the type of career each participant has, family research may ask for marital status and number of children, and political surveys may want to know your address. Would socioeconomic status or household income be useful? With college class evaluations, how about how many years you’ve been in university? In the end, we advise you to include only demographic questions that are relevant to your research questions and/or hypotheses.

2. Closed questions In the last chapter we talked about the difference between open and closed questions. Open questions give participants freedom to explain rich details about their attitudes and experiences, whereas closed-ended questions limit responses to a categorical question (e.g., What are the main reasons you haven’t left your current job?) or a scaled question like a Likert-type question (e.g., How much stress do you experience at work?). In both cases you are hoping to tie their experience or attitudes to common preexisting attitudes, opinions, labels, or categories. Your literature review should help you choose what makes the most sense for your survey. The goal is to identify and confirm variables so that you can

8: Surveys  125

Categorical question 1. What are the main reasons you haven’t left your current job? (check all that apply)  Strong coworker relationships  High job autonomy  High salary  Ethical management Figure 8.2  Categorical question

compare, contrast, and predict patterns across large samples. For example, you could show that employees in certain professions don’t quit because they like their salaries, but others stay based on their ability to be creative.   Scaled questions are meant to identify degrees of attitudes, for example, how much or how little you like something. One of the most commonly used scaled questions are known as Likert-type scales (see Figure 8.3), which are usually used to test hypotheses. Remember, a good hypothesis will have at least two variables that are related in some way. These variables are “co-related,” or correlated positively (which means that when one variable moves, the other variable moves in the same direction) or negatively (which is when the variables move in the opposite direction). Say you have the following hypothesis: H1: Interns who report strong communication satisfaction with their coworkers will also report high job satisfaction.

What are the two variables? Communication and job satisfaction. What is the relationship between the two variables? Positive, since we think that increasing communication satisfaction increases job satisfaction. The best

126  Part II: Methodology overviews Likert-type scale questions For the following questions select the number that best describes you. Strongly Agree Agree Uncertain Disagree Strongly Disagree 5 4 3 2 1 1. I am very satisfied with the conversations I have with my coworkers. 2. I like talking with my coworkers at the office. 3. The conversations I have with my coworkers are very rewarding. 4. I do not enjoy talking with the people I work with. 5. I hope to have more conversations like the ones I have with my coworkers. Figure 8.3  Likert-type scale questions

way to measure this in a survey is to find scaled questions that represent both communication satisfaction and job satisfaction. In Figure 8.3 we’re using an existing scale on “communication satisfaction” to figure out their degree of satisfaction on a 1–5 scale. Similarly, you would present a Likert-type scale for job satisfaction, and then in your data analysis see how they are (or are not) related to each other (see Chapter 11 for how to figure out those relationships).   Another useful form of scaled question is called the semantic differential question. Instead of a 1–5 scale from strongly agree to strongly disagree, the question has two polar opposite terms on each side of the continuum. Participants are asked to select how they feel within that scale. This type of question can be clearer than a Likert-type question and allows participants to respond more accurately in the middle (see Figure 8.4).

8: Surveys  127

Semantic differential question Your romantic partner is Safe ____ ____ ___ ___ ____ Candid ____ ____ ___ ___ ____ Straightforward ____ ____ ____ ___ ___ Respectful ____ ____ ____ ___ ___ Considerate ____ ____ ____ ___ ___ Honest ____ ____ ___ ___ ____

Dangerous Deceptive Tricky Disrespectful Inconsiderate Dishonest

Figure 8.4  Semantic differential question

Validity and reliability If you’re writing questions, particularly if you’re going to test a few hypotheses, you might ask yourself: “why do I need to find existing scales? Why can’t I just write my own questions to measure the variables in my hypotheses? Can’t I just write a few questions to get at how satisfied someone might be in a specific context?” Well, yeah, maybe. You can write out five questions on a Likert scale. And those five questions might look like they’re addressing satisfaction, but you might be wrong and have wasted the survey. People also often ask why we don’t just ask one question instead of five. Well, it’s not that easy. As perfectly as you may have worded the question, people may misinterpret it, rush through it too quickly to fully understand its intent, or simply click the wrong box by mistake. It’s always better to include a few questions, worded similarly, and then take the average score to get the most accurate response. Think about your college classes. What if the entire grade depended on one quiz score? Would that be an accurate depiction of how you are doing in the class? Probably not.

128  Part II: Methodology overviews

Figure 8.5  Validity and reliability

The only way to know if you have good, useful questions is by testing them for validity and reliability. Reliability refers to how consistent each question within your scale is with the other questions. Examine the communication satisfaction scale again. Notice how similar each item is? A great scale will have a few questions that ask about the variable you want to measure, no more, no less (see Chapter 11 for how you can test this). Validity determines whether or not the questions you are asking actually represent the things they should. Do they represent the variable you intend?

8: Surveys  129

If you think about your questions as a game of darts, validity would be the darts that surround the bullseye (closely reflect the variable you’re testing). Reliability would be how consistently you throw those darts. Take dart board #1 in Figure 8.5. Each item within that scale is very similar with one another. They are all assessing the same variable, but it might not be what you intended to measure. The darts on dart boards #2 and #3 are all over the board, which means the questions are so different that it’s almost impossible to determine if they mean what you intend. Dart board #4 is what you really want. The questions are measuring what I want, and responses are all consistent.

A NOTE FROM YOUR FRIENDLY NEIGHBORHOOD LIBRARIAN: How might I track down existing, valid, and reliable scales? While you can go it alone and make your own questions, it’s usually better to use ones that are already valid. Many disciplines have established and extensive scale source books that you can use, and we’ve linked to some of these in the online resources. You can also look at articles that are doing something similar to your project, and draw from their survey scales. Just make sure they’ve chosen valid, reliable scales first! This process can be a bit lengthy, so you might decide just to search for these scales themselves, rather than searching for articles that contain them. To find useful scales you can do a couple of things: you can go to

130  Part II: Methodology overviews

the professional associations for the field in which you’re researching, such as The American Psychological Association (APA), who includes these scales in their database PSYCHtests, the American Sociological Association for Sociology, Rubin, Rubin, Graham, Perse, and Seibold’s (2009) sourcebook or the National Communication Association for Communication, and Bearden, Netemeyer, and Haws (2011) for marketing. You’ll also notice that a quick Google search will help you find some freely available scales, as well as bring up some titles of some physical and ebook titles in your respective discipline that you can then search on your library’s catalog. Some LibGuides are specifically devoted to scales (which should come as no surprise at this point!). Lastly, if you find a particularly decent-looking scale sourcebook and your library doesn’t currently provide access, you should request it via ILL.

Open questions Not all questions can easily be evaluated in terms of reliability and validity. If your research project has been framed around Research Questions instead of Hypotheses, you are probably interested in exploring people’s experiences in more depth or richness than a closed-ended survey could describe. In this case, you should use open-ended questions that hope to capture thoughts, feelings, and/or perceptions. You aren’t trying to confirm how two or more variables are related, in this case, so much as you are hoping to find connections between influential themes. While you can do this with in-depth interviews, what if your topic is very sensitive? What if you’re asking about instances

8: Surveys

131

of abuse, or the stigmas against poverty or being bullied? In these cases, while an open-ended survey might not be able to probe for additional information, because social desirability bias will be reduced dramatically participants may be more inclined to give rich and honest replies.

Common survey question mistakes If you’re writing your own open-ended questions to gain more information from your participant, be careful not use any of the three most common survey question mistakes:

Bad question example

Why are these bad?

Corrected

Double-barreled questions: “What do you value most about your job and why you do you remain in this line of work?”

By asking two distinct questions at once, you run the risk of losing valuable responses. Instead, ask them separately.

1. What do you value most about your job? 2. Why keeps you at your current job?

Leading question: “Would you agree that you are a team player at your organization?”

Leading questions direct the participant to answer in a certain way. Participants tend to respond to what they think the researcher wants to hear and not always honestly. Instead, remain neutral.

1. What does it mean to be a “team player” at your work? 2. In what way, if any at all, have you exemplified this?

Figure 8.6 Bad questions

132  Part II: Methodology overviews Loaded question: “How have you resolved unethical dilemmas you’ve encountered?”

Loaded questions trap your participant. It assumes action has already happened and are guilty regardless of how they respond. Instead, use a prompt.

1. Have you ever witnessed unethical behavior at your work? 2. If so, what actions did you take to resolve it?

Figure 8.6  Continued

Survey flow and design Now that you’ve written your demographic questions, posed a few categorical questions to determine differences, implemented a few existing Likert-type scales to assess relationships between variables, and developed a few open-ended questions to explore nuances, you need to step back and take a look at your survey’s layout and design. It should be extremely clear and easy to follow both in flow (is it logical? Is it overwhelming?) and in question wording (do they know what I’m asking? Will this give me useful data?). Keep in mind that your participants in many cases are either being incentivized or required to take your survey. So, a good survey should flow well, be written in plain language, and have a simple format. There’s a delicate balancing act here, though, between a strong, reliable survey and an exhausting survey. “Survey fatigue” is real, and you don’t want your participants to get so tired of taking it that they stop halfway through or just give random answers to finish more quickly. Some course evaluations are so long that students simply don’t have the energy to write in the more open-ended questions that most professors actually value and read, for instance. So then the survey isn’t accomplishing what it set out to do.

8: Surveys  133

Survey types Although it may seem like a simple issue, you’ll need to decide on the best way to deliver the survey. Face to face? Phone? Mail? Online? The decision isn’t as simple as it seems, because sampling methods and survey types go hand in hand. For instance, if you’re using a convenience sample of people in your store, you might go old-school with paper, pencil, and a clipboard, or you might add a brief survey at the end of their receipt or transaction at the register’s iPad. Face-to-face surveys like this are useful because the data is gathered immediately and you do not have to worry as much about “user error.” However, as with interviews they can be time consuming and can be heavily impacted by both interviewer and social desirability bias. Phone surveys are still used by large corporations who are committed to probability sampling and hoping to report on public opinions. You may have received a call from Gallup about which political candidate you are voting for, or connected a Nielsen Media box to your cable that monitors which television shows you’re watching. Mail surveys suffer from low participation because it’s easier to throw away junk mail than it is to hang up on someone, but the cost is low and you can incentivize participation with coupons, gift cards, and even cash in envelopes. Today, though, most surveys are online surveys because they can easily be created, changed, shared, and modified, and some allow for including multimedia content or even conversations between participants.

IRL: COLLECTING DATA AT WORK Most smart professionals are constantly looking for skills that they can put on their resume and mention in future job

134  Part II: Methodology overviews

interviews. “Data analytics” is one of the most sought-after skills in the current job market, and a research methods course teaches you how to do it strategically and effectively. Say your boss is looking for ways to increase revenue, decrease employee turnover, and boost overall company morale. You hear your supervisors talk about this in nearly every meeting, but no one seems to have any answers. You might mention to your boss, “You know, I’m not exactly sure what the solution is, but I can certainly put together a survey with some common reasons why people might stay and leave their place of work.” Then, you could use SurveyMonkey or Qualtrics to ask employees questions on (a) employee engagement, (b) job satisfaction, (c) burnout potential, (d) and coworker satisfaction and assess where everyone’s attitude is on these scales so you can see where the problems are. You could reward them with a free lunch and a raffled grand prize for participating. After you get the results, you could hold workshops or even hire a consultant to fix the issues they identify. Congratulations, you just paved the way to your next promotion.

For further reading Bearden, W. O., Netemeyer, R. G., & Haws, K. L. (Eds.). (2011). Handbook of marketing scales: Multi-item measures for marketing and consumer behavior research (3rd ed.). Thousand Oaks, CA: Sage. Rubin, R. B., Rubin, A. M., Graham, E., Perse, E. M., & Seibold, D. (2009). Communication research measures II: A sourcebook. New York, NY: Routledge.

CHAPTER

9

Observational methods in practice (participatory-action research, experimental research, theory, and metasynthesis)

OVERVIEW AND OBJECTIVES This chapter is the last of our methods chapters and gives a brief overview of a few different methods that are each based primarily in observation. Participatory-action research (PAR) focuses on working directly within communities to improve their conditions, and is often a mix of other methods to preserve a scholarly approach while not separating the scholar from the community. Experimental scholarship contrasts “control” and modified groups within highly controlled environments to see how changing a variable might impact a system, process, or interaction. Theory, broadly defined, is a way to observe your surroundings and then draw from existing knowledge, logic, and intuition to come up with explanations for how things work and what they mean. Metasynthesis is a sort of highly augmented literature review, where you bring together a variety of sources on a topic and show how looking at them all at once provides some interesting insights.

136  Part II: Methodology overviews There are many ways to do research, as we’ve seen. Different researchers simply think about their work differently. Is there anything wrong with someone saying they want to know how people think about their sex lives, and want to understand it in as much specificity and “certainty” as possible? This can certainly reveal many unanticipated results, as Alfred Kinsey found out in the 1940s. But that data isn’t necessarily any more “accurate,” or more “useful,” than someone sitting in their living room thinking about their personal experiences and imagining the structures and concepts that gave rise to the various myths and preconceptions that create our understanding of what “sex” and “gender” even are. Similarly, does conducting twenty focus groups tell you much more than working intensively with an organization that helps counsel teenagers struggling with their sexual identity? While we can’t go into every possible research method in this book, since that’s not really the point, we do think it’s important to give you a brief overview of a few other kinds of methods that are grounded in observation.

Participatory-action research The first of these methods, Participatory-action research (PAR), is similar in some ways to ethnography, which we addressed in earlier chapters. Both are a form of “participant observation,” which means that the researcher is both observing and a part of what they are observing. It doesn’t matter whether you want to be a part of it or not; simply the act of observing a group changes that group if they’re aware of it. However, while in ethnography the researcher is present but usually doesn’t attempt to interfere or change the relationships or situation that they are observing, PAR researchers argue that because simply observing changes things, it is perhaps more ethical to actively intervene to make the world a better place. PAR is primarily involved in work with community groups, therefore, and researchers see

9: PAR, experiments, theory, and metasynthesis  137

themselves as active collaborators. Through interaction and discussion they form action and activism plans to positively impact that community. While this always runs the risk of the researcher pushing their beliefs onto the community if their experiences, attitudes, or beliefs differ too widely from the participants’, the intent is to embody the change that critical scholars often claim is central to their work (rather than to only write about change). Central to PAR is collaboration, not only in action but also in making meaning and defining problems (and solutions) themselves. For instance, a scholar working on poverty would often discuss the conditions with the community to decide as a group what can and should be done to create change. This might be fundraising, but it might also be organizing labor groups, fighting predatory businesses, creating community gardens, or any number of other solutions. In other words, PAR doesn’t assume it knows the solutions. Like other forms of research, in its ideal form it is guided by the data, not by the preconceptions and desires of the researcher. Moreover, it attempts actively to empower the subjects of the research by making the process of research a collaboration that is grounded in the needs, desires, and agency of those subjects. The line between activist, advocate, volunteer, consultant, ethnographer, and researcher more broadly defined are blurry here, but intentionally so. Outcomes from PAR could be a policy or plan to present to the local Board of Education, an infographic flyer to teach people how and why to use condoms, or a cookbook or cooking classes for single parents to create inexpensive, nutritious meals in a short amount of time. The key here is to remember that this comes from a conversation with the community about priorities and desires. What do they want, what do they need, and how? How can they be involved and empowered in the process, and how can you help them as a servant leader rather than as a voice of power and authority? Example: PAR is often not a method that we think of connected to the job market or professional development. However,

138  Part II: Methodology overviews it can be a very powerful tool and useful skill. If you’re working on corporate social responsibility, for instance, you’re trying to figure out how to make your company do something to benefit both itself and the community. You might start by looking at how the various stakeholders within your organization think of their lives and their relationship with the company, and the struggles that they encounter with both. For instance, in one of Ted’s classes, students were assigned to create a public relations campaign to make the community better in some way. One student was very involved with our Sexuality, Women’s, and Gender Studies program, and she spoke to her community about the challenges they face. Rather than emphasize pay equality, maternity leave, or other large social issues women face, however, she chose a different approach. What she heard again and again from women related to menstruation, particularly that they worried about not having easy access to pads or tampons. She found that this led to frustration, shame, and anxiety. She also found through her literature review that while women often feel disconnected from their employers, a perception that the employer was thinking of them and their needs increased job satisfaction and retention. Therefore, a strategic intervention would be to place baskets of free tampons and pads in every women’s bathroom. The best part was that this was incredibly inexpensive, and yet had a high return on women’s lives and the bottom line of the company.

Experimental research Experiments seem like they’re easy to do, since the idea isn’t that complex. Take some of one thing and don’t do anything to it. Then take some of the same thing and do something to it. Compare. Then go eat ice cream and celebrate a successful experiment. In reality, experiments can be one of the more difficult research methods to design and pull off. We’ve already

9: PAR, experiments, theory, and metasynthesis  139

discussed some of the ethical issues with experiments, as well as the necessity for using an IRB, in Chapter 4. What we haven’t discussed in any detail is how to actually do that. The main challenge with experiments is striking the balance between control and reality. With experiments in the social sciences you’re designing a procedure to study humans that must be controlled and standardized for each person. Therefore, potential outside influences are minimized. At the same time, you’re trying to create a scenario that mimics the real world so that you can draw real-world conclusions. In theory, in an experiment you want to see if doing something to a group of people will result in a change. For example, perhaps you’ve developed a potential cure for an illness, and now you want to see if this pill will cure it. In order to find out, you’re going to need two groups of people, assigned randomly (we can’t risk picking these groups ourselves and potentially tainting the data). One group will serve as the experimental group (i.e., the group that will experience the pill, intervention, or stimulus to determine change). The other group will be the control group (i.e., the group that will participate in your experiment but will not be subjected to the pill, intervention, or stimulus). This seems simple enough, but here’s the challenge: both groups need to be tested in the same, controlled, environment. Since people’s experiences in real life are anything but controlled and standardized, you have to create an environment that is the same for every person in your experiment – otherwise, you can’t be sure that what you introduced to them is the thing that is causing the change. Critics of experimental design call this an “artificial setting,” and they have a strong point. Bringing people into a controlled environment (often a laboratory or an office staged to appear like “real life”) isn’t real life. It is artificial. So while you are doing your best to make sure that every person in your sample is experiencing the same environmental conditions, you are also creating an artificial world that may not

140  Part II: Methodology overviews reflect real life findings. Nonetheless, experiments are a great way of isolating whether or not a specific stimulus is causing a change to your sample. Example: We couldn’t resist using a famous study that Josh uses in his class. This study examines the connection between kissing and allergic reaction reduction (because science, right?). In this experiment the researcher wanted to know whether the act of physical intimacy, specifically measured via kissing, could reduce allergic reactions in the body. So how would you design that experiment? And how would you accomplish this in a way that could confirm that it’s the kissing that is reducing the allergic reactions? Dr. Kimata began his experiment by randomly assigning skin allergy sufferers into two groups: an experiment group and a control group. The allergy sufferers from both groups were instructed to pair up with their romantic partners for this experiment. He used “kissing” as his manipulated variable and then measured the degree of change in participants’ IgE levels (levels of an antibody produced by the immune system during allergic reactions). Because the outcome of the experiment hinged on the degree of change in patients’ IgE levels, he had to take samples before and after the couples “made out.” At this point, you probably have a few more questions. Where are these people kissing? Is he near them? And how do we know that it’s the act of kissing that’s changing the IgE levels, and not merely physical touch or even close proximity to their partners. Great questions. One of the biggest strengths of true experiments is that each one has a control group. Dr. Kimata had all of the participants undergo the exact same conditions. Both groups came into a low-lit room with soft music. The experiment group was instructed to kiss for thirty minutes, while the control group was told to sit next to each other, under the same conditions, and embrace each other for the duration of the experiment. Subjecting a control group to the very same condition, except the manipulated variable (kissing), can help ensure that there aren’t

9: PAR, experiments, theory, and metasynthesis  141

other factors affecting the outcome. As for the artificial setting, there’s no great way around this. Researchers using experimental design believe that the pros outweigh the cons here. Sure, it would be interesting (and absolutely creepy) to see if the results would change if IgE levels were taken before and after an organic kissing session within the couple’s natural environment. But this would be very difficult to compare across your sample. What if some couples had music and others did not? What if some had short kissing sessions, while others had longer ones? What if some didn’t stop at merely kissing? We think you get the point. Standardized, artificial settings are the only way to try to silence the effects of other possible variables contributing to the story. In the end, Kimata’s results were confirmed. The kissing group’s IgE levels were significantly lower than the control group levels, and so the act of kissing was shown to reduce allergic reactions in his sample. Interesting, right?

Metasynthesis This method is, like PAR, a relatively new approach to scholarship that attempts to address shortcomings in the range of methods researchers use. In this case, it isn’t involving the researcher in the detailed activities of a community. In fact, while it is often qualitative, it is a broad rather than narrow method. In some ways, you could say that a metasynthesis is a discourse or content analysis of the scholarship about an issue. However, while a metasynthesis often looks a lot like a literature review, it’s very different. Instead of just surveying previous studies and organizing them, it looks deeply at the data used by those studies and brings them together into a new, larger whole. This can be very useful, since individual data sources and samples might produce very conflicting results, as can different methods. Bringing together disparate approaches to the same topic can reveal new

142  Part II: Methodology overviews connections, inconsistencies, and trends that wouldn’t be visible through a small set of studies (if you think back to the “blind men and the elephant” metaphor we mentioned in Chapter 3, you’ll see what we mean). Example: Because research on businesses doing good for the community is so inconsistent, it can be a big challenge for employees or even CEOs to advocate for it. Shareholders (or bosses) often want to know how much money it will return to the company, or if it is a “waste” of money. A metasynthesis can really help with this. By gathering all the various forms of data from the hundreds of studies done over the past couple of decades on corporate social responsibility (CSR), a researcher can more accurately point to why the results from previous research are so disparate. They could then suggest that the company duplicate the more successful methods with their own products and their own intended customers or employees to determine impact. After all, even retaining a few employees could save the company hundreds of thousands of dollars, so you don’t necessarily need to show a huge impact to demonstrate that CSR is worth doing. The important thing is that you can show that you’ve done your study using the best practices for researching CSR, so that your boss or your company’s shareholders have confidence that you’re right about the impact.

Theory Some of the most impactful research ever conducted was done by people who sat in their homes, offices, or even in parks and libraries. It didn’t necessarily involve data the way we normally think about data. It didn’t necessarily involve “participants” or “respondents,” and wasn’t “repeatable” the way we often think research should be. Instead, it is the product of deep and considerable thought, discussion, evaluation, editing, and a mixture of

9: PAR, experiments, theory, and metasynthesis  143

ethics, logic, and emotion. “Theory,” as we often call this form of research, doesn’t need to be proven to be powerful or influential. In fact, it might not be possible to “prove” theory within the current limits of society, science, or technology. But without it, we wouldn’t know that the Earth is round, the stars are massive balls of energy and matter, that gravity and time are related, or that light is both a particle and a wave. We wouldn’t have computers, and we wouldn’t have philosophy. We wouldn’t be able to conceive of a world without war, hunger, or abuse. We wouldn’t have dreamed of democracy, or a free press, or electricity. In some ways, you could say that theory is philosophy. It takes observations and conversations about the world and processes them through the human mind and the social imagination, creating ideas about how things are created, organized, changed, and disrupted. Like most philosophy, it takes years of study to be able to produce theoretical work that is grounded in true understanding; at the same time, like philosophy it can often seem like common sense, especially after the fact. But like a painting by Jackson Pollock, just because something can be done doesn’t mean it has been done, and certainly not with the same inspiration or intention. Theory, like all research, adds to a conversation and takes it to the next level. It is not a method we recommend to researchers in the beginning of a project, but at the same time it is something you shouldn’t ever avoid. Even the questions you have, and the choice of your method, is in many ways already creating theory. Once you have those ideas, however, at least in early stages, we recommend that you let data guide you, and not just your ideas in and of themselves. Example: The early days of CSR were very different than many approaches taken today. A car company might have donated to the local ballet, or hosted a 5k race to promote breast cancer research, or bought expensive paintings for a museum. Each is considered to be CSR, but these approaches were often

144  Part II: Methodology overviews not connected at all to what the company did or stood for, who its employees were, or who their customers were. Some theorists then suggested that CSR wasn’t inconsistent in impact because it wasn’t strategic. In other words, the problem wasn’t CSR itself. Theorists like Coombs and Holladay argued for “strategic corporate social responsibility,” and others like Porter and Kramer argued that we think of CSR as “creating shared value” between a company and a community. These arguments weren’t focused on data; they were focused on ideas, and in many ways they reframed both what was studied and how.

For further reading Kimata, H. (2003). Kissing reduces allergic skin wheal responses and plasma neurotrophin levels. Physiology and Behavior, 80(2–3), 395–398. Kimata, H. (2006). Kissing selectively decreases allergen-specific IgE production in atopic patients. Journal of Psychosomatic Research, 60, 545–547.

Data analysis and the final push

PART

III

CHAPTER

10

Analyzing qualitative data and making sense of detail

OVERVIEW AND OBJECTIVES This chapter goes into more detail about how you analyze qualitative data once you’ve gathered it. Focusing on textual analysis and interviews/focus groups, we discuss how you determine what to look for, and how to choose between a tightly controlled and structured approach and a loose, open one. It’s probably not a shock that we suggest something in between, a “managed” approach that draws from existing research and your research questions rather than hoping that the data will inherently guide you (or that it will do so without bias). In each case we suggest that you look at what might be “silent” or “erased” material rather than only what you see. Finally, we give you some hints about how to make this process easier for yourself as well in the way you go about it and by getting some help from a computer program.

10: Analyzing qualitative data  147

While we’ve given you an overview of many different methods throughout the book, we haven’t discussed in detail how to turn your raw data into anything useful. In other words, how do you break it all down? In some ways we’ve prepared you for this even as far back as the “choosing a method” and “writing a literature review” chapters. What you were doing there was actually a thematic analysis of your project, figuring out what was important, what wasn’t important, and how you should structure your argument as a conversation with other research. This is also how you analyze almost all qualitative data. You look for themes, structure the data into those themes, and then talk about how the data fits into or interacts with the themes. This can be a very organic process by which you allow the data to take more control, or you can “manage” it by placing it into existing structures, often called “typologies.” Both have their benefits, and both have their problems.

(Managed) textual data (often used for textual, content, and discourse analysis) In the project on 13 Reasons Why, it was possible to watch the show and see what popped into our heads. Those could have been aesthetic features, scripted dialogue, settings, plot, and other elements. Some of the best textual data is organized in that organic way, because it allows you to easily discover elements that you might not see otherwise. For instance, in one student’s project on the show Big Little Lies, she wanted to study politically progressive images of women in the show. However, when doing an organic content analysis of the first season she realized that the images weren’t quite as progressive as they seemed. While women often appeared to dominate the storylines, imagery, and narrative, that was with several qualifications. In some cases their strength was also connected to psychological instability,

148  Part III: Data analysis and the final push modified by abusive relationships, or contained within particular settings. In fact, when women were seen to have any sort of power, it was usually away from the home and in a natural setting like the beach. In other cases women were primarily seen as mothers taking care of their children; their “empowerment,” in other words, was almost always qualified by other elements. However, it never would have occurred to us to code for natural vs. human-made settings until seeing it come back as a trend during the process of the content analysis. Organic textual readings allow the data to modify the typology itself. There’s nothing wrong with having a clear typology, either. When doing the 13 Reasons Why project, it made sense to look for elements drawn from best practices regarding effective treatment and conversation with people at risk for suicide, because that’s what we were looking for. In another project where we looked for successful web development for nonprofits, it made sense to look at what the scholarly research and the discussions with industry professionals about best practices had to say about success or failure. This adherence to a typology sometimes means that you limit yourself to that data exclusively. However, it’s also possible to remain open to other possibilities, and you can always code for additional data. The key is to remain critical of your own work, and aware that neither you nor previous scholars have finished the project; otherwise, why would you be doing it? There are many ways to actually do this, but the way we often recommend is to make a spreadsheet in Excel. Yeah. We know. But trust us on this one, it’s going to make your life a lot easier. Imagine you’re doing an analysis of social media posts. What will you code for? Some of this will come from your literature review, and some is just logical depending on your research questions (see our suggestions in Chapter 8). For instance, you’ll want general information like the platform, the date and time it was posted, how many views, shares, and likes it received, and how many comments there were. Great. Each of those is a column in

10: Analyzing qualitative data  149

the spreadsheet (we’re up to seven now). How about if there’s an image, a video, or just text? That’s either one more column or three more (we suggest three, so now it’s ten total). How many words are written? How many hashtags? Is there a clear “call to action” like a challenge video or a request to donate? Is there a link for further reading? Is there some engagement like a poll, quiz, or game (we’re at fifteen minimum now)? What quality is the image (low, medium, high), and does it look like it’s taken by them, or is a stock photo (seventeen)? Are there people, and if so how many? What tone do they use (happy, sad, playful, angry)? Is there an ethical appeal, a logical appeal, an emotional appeal (twenty-two minimum)? What race or ethnicity do the people seem like they are, if there are any, and what gender (twenty-four)? How many of each type (twenty-six)? Do they tag or reference anyone else, including maybe another organization or influencer (twenty-eight)? This gets long, but it becomes an incredibly efficient and rapid way to analyze and structure your analysis. When you make each code, figure out what kind of code it is. Is it for the presence or absence of something (1 or 0)? Is it how strong/forceful/emphatic it is (scale of 0–3, maybe, 0 being “not present” and 3 being “strongly present”)? Or is it a category like “young/old,” “product/event/news?” (You can even make Excel give you a drop-down menu so you don’t have to type each choice every time. Check out our online resources for a guide.) Is it a number like “120 words?” Once you have the spreadsheet with your codes, you can do your analysis. What you’re looking for are correlations. You do this by searching for anything that looks different, or anything that looks like it happens a lot (one way to do this is to use “conditional formatting,” to automatically highlight things in different colors, and the online resources show you how to do that). But it doesn’t have to be fancy, either. Excel will automatically generate charts if you highlight a couple of pieces of data (see our online resources again), but if you use a couple of simple tools you can get a sense

150  Part III: Data analysis and the final push of how different sets of texts compare. For instance, if you want to know how many people on average appear in two organizations’ Facebook posts, you can use an AVERAGE(FirstCell:SecondCell) formula for each organization. Similarly, that would measure how negative (-5 to 0) or positive (0 to 5) a politician’s rhetoric is, or how many sentences out of the entire speech were positive or negative (you’re coding for a number there). You can also add up how many posts have a call to action, or how many use an emotional appeal. Once you do that, of course, the correlations become much easier to see (you will see how to analyze this in further statistical depth in Chapter 11). While content analysis and discourse analysis do sometimes rely on correlation and numbers, it’s also usually important to discuss some examples of each of the trends you’re seeing. These often take the form of miniature textual analyses, just so your reader understands what exactly is happening with various trends. It doesn’t take many; one or two examples (what we call “paradigmatic examples”) is usually enough to give readers enough of an idea that they understand your point. Remember that sometimes you might be looking for low numbers as much as high, or you might look for things that are “outliers” (posts that are very different). For instance, wouldn’t it be interesting if the only posts that show someone working in manual labor show people of color, or men (so White or Women would be coded as “0”)? Wouldn’t it be interesting to see why two out of thirty posts went viral and got over one million views, in comparison to the other posts? What changed? This is really where you get to the “managed” concept. While a textual, content, or discourse analysis might seem to be totally organic, usually we’re choosing to focus on certain data over other data, and connect one category or code to another. The reality is that all research is managed at some point or other, but you should try to keep in mind that the amount of management, and how you manage, is really important. After all, your

10: Analyzing qualitative data  151

preconceptions or biases might send you in a bad direction. For instance, one student working on images of the police during a time of intense protests against police brutality performed a content analysis of news segments but didn’t notice that a great deal of her data didn’t actually fit her desired outcome (to show that the police were unjustifiably portrayed negatively), or that she wasn’t coding for things she needed to (that the victims were portrayed as thugs or shown with negative images, and that protesters were often characterized as “rioters”). Once she realized that was happening, the entire project, and her conclusions, changed completely.

(Managed) grounded theory (often for interviews, focus groups, and surveys) This is even more important, and yet sometimes harder, for research on participants. As we discussed in Chapter 7, bias is inherent and inescapable, especially when you’re working with people. There are so many variables that it’s far more difficult to address them all effectively. We mentioned that video and/ or audio recording participant research like interviews and focus groups is a helpful way to approach it, especially if you have more than one researcher analyzing the data. Transcriptions are also extremely useful, since you can remove issues like accents, skin color, and paralinguistics (e.g., that women in the United States speaking English often end sentences with a nonverbal “question” sound that might reduce their forcefulness or credibility with some raters, or that Chinese women often have a distinct voice from men in the way tones are used for the words). But what do you do when you have the transcriptions? How do you actually analyze what people say? In reality it’s not that much different from doing a literature review or the types of textual analysis we discuss above. The data

152  Part III: Data analysis and the final push can guide you here, as can any hints for themes that you might find in the literature to help you out. For instance, as we discussed above, in Owen’s (1984) article on thematic analysis he suggests looking for repetition (a participant that says the same thing multiple times), forcefulness (the strength of a participant’s statement, which might or might not appear with other participants), and recurrence (when several participants say the same or similar things). This framework is really useful, because it allows you to eliminate even more issues of bias. In many ways you’re looking for both similarities and dissimilarities, and that gives you a lot of power as a researcher to make sure you’re doing the data (and your participants) justice. The process of pulling out themes from the data, rather than choosing themes and then laying them on top of the data or forcing the data into them, is called “grounded theory.” A more structured approach, like we saw above, is called “managed grounded theory,” in which you have an idea of themes before you start but you remain open to them evolving or even being discarded as you look at your data. Rather than a spreadsheet, however, we recommend that you take all the transcriptions (and we can’t emphasize enough how important it is to transcribe! See our online resources for some suggestions on how to make that faster) and highlight them according to theme. This looks a lot like the way we talked about doing literature reviews in Chapter 3, right? That’s no accident. It really is the same process. Find themes as you go. Anything that stands out, that you see repeated within or across transcriptions, or moments when you ask yourself “wait, but what about . . .” are things to code for, even though in the last bit you’re just pointing out the absence of something (an emotion, a logical connection, etc.). Then you label each highlighted section with the participant number or pseudonym and place all the highlights of each color together. Those groups of quotes then become the basis for your analysis.

10: Analyzing qualitative data  153

The fun thing about this kind of data analysis is that you can allow the participants to speak, just like your paradigmatic examples “speak” about your data in text-based analyses. This kind of research tends to use a lot of paraphrases and a lot of quotes, sometimes even extremely long quotes, to make their points. You’re not necessarily trying to say what these mean, you’re simply showing in a structured way how participants make sense of their reality, how they take in and then express various stimuli like a TV show, conversation with a boss, or relationship counseling appointment. That doesn’t mean you aren’t being persuasive and making an argument; you are. It’s just that the data speak for themselves, and if you’re being ethical, they speak accurately. Remember, these have a relatively small N; you’re not interviewing thousands of people in most cases, so you have very little ability to generalize. But you can certainly say that these people felt these ways about the topic, which can be just as powerful and accurate as a survey with thousands of responses.

WORK SMART, NOT HARD: USING TOOLS LIKE NVIVO FOR ANALYSIS Remember, with qualitative analysis, you’ve signed up to grapple with rich, detailed, complex data. Whether it’s over twenty hour-long interview transcripts or hundreds of newspaper articles, you probably have a ton of information to sort through. Luckily, there are a few software programs that can make your life a bit easier. NVivo, a qualitative data analysis software package, is one of the most widely used tools in helping researchers create coding schemes and themes for large amounts of data. Not unlike the “search toolbar” on most websites, NVivo allows you to create your

154  Part III: Data analysis and the final push

own coding categories and search conditions for those categories. Let’s say you are analyzing the portrayal of emotions via character dialogue across several screenplays. Yes, a simple solution would be to create searches for (a) “anger,” (b) “sadness,” (c) “jealousy,” and so forth. But think about how many different words and phrases could be used to portray the same emotion? If you only searched using the specific word for each of the seven primary emotions, you would miss synonyms and similar phrases that could be included within your coding categories. Nvivo allows you to build onto your categories by including phrases, terms, slang, and other words that represent your specific categories. Once the Nvivo program tags each of the phrases and aligns them within your coding scheme, you can further subdivide your categories as you see fit. This can even be done with images and video, and you can share it among collaborators for a smooth process.

Silences We think it’s worth emphasizing that one of the most important things to be aware of as a researcher is the silences in your data. What are your participants not telling you, and why? If you know that statistically 90% of a group of college students is having sex, and yet no one wants to talk about it, why is that? If you know that most of them have at least some sexual interest in their same sex, why do so many of them claim to be “totally straight?” Why would the one or two students of color in a class not speak up about issues that impact them directly, and the experiences they’ve had that prove it? We’re kind of stacking the

10: Analyzing qualitative data  155

deck here, of course. The answer to almost all of those questions is social power, and the stigma that is often attached to speaking out against power. As researchers, we have to remember that this isn’t just about interviews or other forms of human subject research. It’s also in our textual data. How? That seems like it’s counterintuitive, right? The texts and the data from the texts will guide you. But what if there’s something there that you’re not looking for? For instance, “third wave feminism” strongly critiqued other members of the feminist movement for speaking about “women” as if they were this monolithic group, rather than acknowledging that individual women are in fact very different from each other. The road to empowerment and the road to equality often diverge. Women of color experience femininity and oppression very differently than those of the dominant group, and in fact the third wave pointed out that many of the arguments made by feminists at the time simply weren’t as relevant to them (e.g., does having more women in leading roles in the movies matter when you feel unable to report sexual assault to the police?). This is a type of silence that we can find by expanding our view from what we’re looking for to what’s there, but also trying to look for voices and experiences that might be silenced or erased from the texts. The history of society may be written by the dominant, but it’s our job as researchers to challenge what we think we know. As we mentioned in Chapter 7, bias is something we all have, and it’s our job to try to remove it as much as possible. Common sense may seem logical, after all, but different family, school, economic, and social experiences make for very different versions of it.

For further reading Owen, W. F. (1984). Interpretive themes in relational communication. Quarterly Journal of Speech, 70(3), 274–287.

CHAPTER

11

Analyzing quantitative data and making sense of numbers

OVERVIEW AND OBJECTIVES Building on the last chapter’s approaches to qualitative results, this chapter shows you how to analyze number-focused data. We mainly focus on surveys and how you can use the numbers you get from those data. There are a lot of terms in this chapter, but we place them in tables and charts rather than go over them in too much detail. We want you to know above all what they are and why you’d use them. We’ll also go over using graphs and tables, and talk about how you could use tools like SPSS and Excel to make this process faster.

11: Analyzing quantitative data  157

Loving and hating statistics It’s important to point out that all research attempts to report what is “true.” In the last chapter, we talked about analyzing and reporting on people’s qualitative accounts, their written and/or oral communication about particular events. Since we will never fully know exactly what’s happening inside someone’s head, social scientists have to rely on people’s accounts. Qualitative scholars like words and descriptions. Quantitative researchers condense participant accounts to a number. While this removes some great thick description, there are some big advantages as well. If you’ve chosen to collect numeric responses to questions, you’re most interested in finding trends and patterns across a sample. Simply put, numbers are easier to collect in large amounts and much easier to compare than people’s words. That also allows you to turn the data into infographics (visual representations of data, like pie charts, so people can understand them quickly and easily). So if you want to know how every student feels about their college experience, having them report a number on a scale will give you data that can be presented in a statistic. Other questions on your survey could find out if there’s a difference between freshman and senior attitudes. And even more questions could get to the bottom of whether there’s a relationship between satisfaction and graduation rate. In the end, everything you collect with numbers serves these purposes: (1) it describes your sample, (2) it tests for differences, and/ or (3) it tests for relationships. This is all useful, but most people have a love/hate relationship with statistics. There’s something comforting about wrapping our minds around a number that seems to represent a specific, definitive truth. On the other hand, the idea of taking a course on statistics sounds excruciating, and we know that any c­ omplex system

158  Part III: Data analysis and the final push is extremely difficult to fully describe by only using numbers. After all, 42% of all stats are made up on the spot. Or is it 76%? We forget, but you should trust us anyway because we’re wearing suits.

Descriptive statistics: who are your people? What are your variables? The most common way people use statistics is to describe (1) their sample (e.g., demographics) and (2) what the people within their sample have said and/or feel. The two most common descriptive statistics you might use are percentages and frequencies. It might be important to know how many people in your sample are first-year students vs. graduating students, for instance. It also might be important to know what percentage of people are employed and unemployed when trying to figure out what to do about homelessness, or what percentage feel that everyone is entitled to healthcare, shelter, and food. We’re certainly pointing out the obvious here, but when reporting your sample’s demographics, relaying the percentages of who they are and any other pertinent identifiers, roles, categories, and any checkbox questions help tell the story of your sample to your readers. Frequency is how often each of those percentages occur, which is useful when you have more than two options (year in school, for instance, rather than graduate vs. undergraduate students). Describing variables Let’s say you want to tell your audience how satisfied a group of people feel about a product. When you report this, you are relaying your sample’s “bell curve” for that particular variable, otherwise known as a “normal distribution” (see Figure 11.1). A bell curve is often represented by the following figures:

11: Analyzing quantitative data  159

Figure 11.1  A “normal distribution”

• Mean (M or µ): the average score of every single response (e.g., average grade in class is a B). • Median (Mdn): the middle score if they were lined up from lowest to highest value (e.g., of twenty-one students, what’s the eleventh highest/lowest grade?). • Mode: the value that is most often reported (e.g., most people got Bs) • Number of cases (N): the number of units (e.g., number of students) analyzed. • Range: the field of values from the lowest to the highest (e.g., grades range from D+ to A-). • Standard deviation (SD or σ): how all of the scores are distributed across your sample in comparison to the mean (e.g., were most grades in the B range, or were they mostly As and Ds?). These numbers need to be seen together, since each tells part of the story, but not all of it. I’m sure you’ve combed through

160  Part III: Data analysis and the final push a few amazon customer reviews before purchasing something, right? If the average (M) score is 4.5/5, this tells you that, on average, people were generally very satisfied. But what if only three people (N) reviewed the product? Would you still trust that number? What if out of a thousand reviews the average is a 3, but all reviews are either 5 stars or 1 star? That might make you look a little closer to see if there are stories about it breaking in the first month, or about bad customer service, right? This becomes especially important when your range is wide. In many ways, the mean alone only tells half the story. A large standard deviation communicates that the sample’s scores were scattered greatly from the mean, while a smaller standard deviation tells you that the scores were closer together. Reporting age is a great example of why SD is important. You might list the mean age of your sample as twenty-five, with a range between eighteen and seventy-five. But it’s your SD score that will tell you how spread out (or close) the ages of your sample are. For example, Josh worked on a research project about long-­distance romantic relationships. His research team noticed that the mean age of the sample was twenty-seven years old, with a range between eighteen and sixty-five. Further examination showed that there was fairly large standard deviation, which told them that there was a wide range of ages, instead of a large cluster of twenty-seven-year-olds in his sample. While we’re discussing the best way to report your data, remember that a “normal” bell curve assumes that your score distribution is just that: normal (i.e., the curve is completely symmetrical, which means that the number of scores below the mean are equal to those above the mean). Sometimes, though, you might have a skewed distribution, either negative (where most data is lower than the average, as you can see in Figure 11.2) or positive (where most data is higher than the average, as you can see in Figure 11.3).

11: Analyzing quantitative data  161

Figure 11.2  Negatively skewed distribution curve

Figure 11.3  Positively skewed distribution curve

If your curve is in fact a normal bell curve, you should be fine to just report the mean score. Because of the potential in skewed distribution, though, it’s a good idea to relay the three central tendency measures (mean, median, and mode) to give your

162  Part III: Data analysis and the final push reader a more complete picture of what’s happening. Reporting the median and/or mode can be useful if a few of the values in your data set are extremely high or low and, therefore, will throw off your average. Let’s say you wanted to report how far people travel to and from work every day. You’ve got five hundred employees, but two of them reported thousands of miles of average commute time due to international travel, whereas everyone else lives in the neighborhood. So what would have been an average of 15.5 miles now shows up as 210.7 miles. In these skewed cases, you should include the median to help clarify the inflated average distance. In another example, the mean temperature of Orlando, FL, is 73℉. That’s a great number for Disney to tell tourists. If you live there, however, you know that summers can easily get to 93℉, which is really hot. In this case, a mode of 82℉ is more representative of an average day. In other words, in Orlando more days of the year have a high of 82℉ than any other temperature. While descriptive statistics are the most common and simplest way to report number data, it doesn’t tell a very complex story. Yes, there’s great value in knowing that 40% of your staff is female, or that 62% of the employees at a company have been there fewer than three years, or that 30% of your sample are reporting workplace burnout, but these numbers don’t tell us anything about the differences between genders, between new hires and experienced employees, or those who burn out and those who don’t.

Finding differences through inferential statistics Statistics that reveal these kinds of group differences and possible influential factors at play are known as inferential statistics. They can help you infer, draw a few possible conclusions, and

11: Analyzing quantitative data  163

for the most part tell a more complete story. Do college seniors know more than freshmen? About what? Are there significant differences in how we pay men and women, and is that all men and women or just those who have children? Does socioeconomic status influence certain outcomes? Does race play a role? Most statistical analyses focus on finding significant differences between groups. We want to briefly walk you through three common tests for identifying differences between groups and variables. But before we do, let’s review a few definitions. • Independent variable: the variable selected by the research that does not change and is not affected by the other variable(s). • Dependent variable: the variable that changes based on movement from the independent variable. • Continuous variable: a variable with a numeric value anywhere between a specific minimum and maximum value (e.g., 1–7). • Categorical variable (nominal): a variable with a specific amount of optional responses, not necessarily in logical order. t-tests and p-tests The t-test was developed in 1908 by William Gossett as a quick way to monitor the quality of Guinness beer. Yep, a hired biochemist used science and statistics to track the quality and flavor of one of the world’s most popular beers. (And you thought stats had no real-world value.) In short, the t-test (t) is the simplest, clearest way to determine differences between two groups. Let’s say you want to determine whether seniors have better communication skills than freshman. A t-test compares the communication skill level averages (dependent variable) among the two groups: freshman and senior (independent variable). If the mean value of the senior communication skill level (4.8/5) is significantly higher than the mean value of the freshman level

164  Part III: Data analysis and the final push (4.3/5), you could conclude there is a difference between the two groups. Remember, the independent variable can stand alone – it doesn’t change – and it always fits into a group or category (nominal) when doing a t-test (year in school). The dependent variable (continuous) is the variable that changes depending on which group you’re considering (in this case, the communication skill score). For a quick tutorial on how to calculate a t-test, see our online resources. But how do you know if this difference is not just random? Is there a strong number pattern to show that these two groups are in fact different? A quick significance level (p) test on a computer program like SPSS or Excel will calculate your numbers to see if what you found could have been just dumb luck (completely random) and what percentage is showing a distinct pattern (see our online resources for how to do this). Most scholars agree that if over 95% of your numbers (i.e., p