Statistical rethinking: a Bayesian course with examples in R and Stan 9781482253467, 1482253461

Citation preview

Statistical Rethinking

A Bayesian Course with Examples in R and Stan

CHAPMAN & HALL/CRC Texts in Statistical Science Series Series Editors Francesca Dominici, Harvard School of Public Health, USA Julian J. Faraway, University of Bath, UK Martin Tanner, Northwestern University, USA Jim Zidek, University of British Columbia, Canada Statistical Theory: A Concise Introduction F. Abramovich and Y. Ritov

Practical Multivariate Analysis, Fifth Edition A. Afifi, S. May, and V.A. Clark Practical Statistics for Medical Research D.G. Altman Interpreting Data: A First Course in Statistics A.J.B. Anderson

Introduction to Probability with R K. Baclawski

Linear Algebra and Matrix Analysis for Statistics S. Banerjee and A. Roy

Mathematical Statistics: Basic Ideas and Selected Topics, Volume I, Second Edition P. J. Bickel and K. A. Doksum Mathematical Statistics: Basic Ideas and Selected Topics, Volume II P. J. Bickel and K. A. Doksum Analysis of Categorical Data with R C. R. Bilder and T. M. Loughin

Statistical Methods for SPC and TQM D. Bissell Introduction to Probability J. K. Blitzstein and J. Hwang

Bayesian Methods for Data Analysis, Third Edition B.P. Carlin and T.A. Louis Second Edition R. Caulcutt

The Analysis of Time Series: An Introduction, Sixth Edition C. Chatfield Introduction to Multivariate Analysis C. Chatfield and A.J. Collins

Problem Solving: A Statistician’s Guide, Second Edition C. Chatfield

Statistics for Technology: A Course in Applied Statistics, Third Edition C. Chatfield Bayesian Ideas and Data Analysis: An Introduction for Scientists and Statisticians R. Christensen, W. Johnson, A. Branscum, and T.E. Hanson Modelling Binary Data, Second Edition D. Collett

Modelling Survival Data in Medical Research, Third Edition D. Collett Introduction to Statistical Methods for Clinical Trials T.D. Cook and D.L. DeMets

Applied Statistics: Principles and Examples D.R. Cox and E.J. Snell

Multivariate Survival Analysis and Competing Risks M. Crowder Statistical Analysis of Reliability Data M.J. Crowder, A.C. Kimber, T.J. Sweeting, and R.L. Smith An Introduction to Generalized Linear Models, Third Edition A.J. Dobson and A.G. Barnett

Nonlinear Time Series: Theory, Methods, and Applications with R Examples R. Douc, E. Moulines, and D.S. Stoffer Introduction to Optimization Methods and Their Applications in Statistics B.S. Everitt Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models J.J. Faraway

Linear Models with R, Second Edition J.J. Faraway A Course in Large Sample Theory T.S. Ferguson

Multivariate Statistics: A Practical Approach B. Flury and H. Riedwyl Readings in Decision Analysis S. French

Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference, Second Edition D. Gamerman and H.F. Lopes

Bayesian Data Analysis, Third Edition A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson, A. Vehtari, and D.B. Rubin Multivariate Analysis of Variance and Repeated Measures: A Practical Approach for Behavioural Scientists D.J. Hand and C.C. Taylor Practical Longitudinal Data Analysis D.J. Hand and M. Crowder Logistic Regression Models J.M. Hilbe

Richly Parameterized Linear Models: Additive, Time Series, and Spatial Models Using Random Effects J.S. Hodges Statistics for Epidemiology N.P. Jewell

Stochastic Processes: An Introduction, Second Edition P.W. Jones and P. Smith The Theory of Linear Models B. Jørgensen Principles of Uncertainty J.B. Kadane

Graphics for Statistics and Data Analysis with R K.J. Keen Mathematical Statistics K. Knight

Introduction to Multivariate Analysis: Linear and Nonlinear Modeling S. Konishi

Nonparametric Methods in Statistics with SAS Applications O. Korosteleva Modeling and Analysis of Stochastic Systems, Second Edition V.G. Kulkarni

Exercises and Solutions in Biostatistical Theory L.L. Kupper, B.H. Neelon, and S.M. O’Brien

Exercises and Solutions in Statistical Theory L.L. Kupper, B.H. Neelon, and S.M. O’Brien Design and Analysis of Experiments with R J. Lawson

Design and Analysis of Experiments with SAS J. Lawson A Course in Categorical Data Analysis T. Leonard Statistics for Accountants S. Letchford

Introduction to the Theory of Statistical Inference H. Liero and S. Zwanzig Statistical Theory, Fourth Edition B.W. Lindgren

Stationary Stochastic Processes: Theory and Applications G. Lindgren Statistics for Finance E. Lindström, H. Madsen, and J. N. Nielsen

The BUGS Book: A Practical Introduction to Bayesian Analysis D. Lunn, C. Jackson, N. Best, A. Thomas, and D. Spiegelhalter Introduction to General and Generalized Linear Models H. Madsen and P. Thyregod Time Series Analysis H. Madsen Pólya Urn Models H. Mahmoud

Randomization, Bootstrap and Monte Carlo Methods in Biology, Third Edition B.F.J. Manly Introduction to Randomized Controlled Clinical Trials, Second Edition J.N.S. Matthews

Statistical Rethinking: A Bayesian Course with Examples in R and Stan R. McElreath Statistical Methods in Agriculture and Experimental Biology, Second Edition R. Mead, R.N. Curnow, and A.M. Hasted

Statistics in Engineering: A Practical Approach A.V. Metcalfe Statistical Inference: An Integrated Approach, Second Edition H. S. Migon, D. Gamerman, and F. Louzada

Beyond ANOVA: Basics of Applied Statistics R.G. Miller, Jr.

Decision Analysis: A Bayesian Approach J.Q. Smith

Applied Stochastic Modelling, Second Edition B.J.T. Morgan

Applied Statistics: Handbook of GENSTAT Analyses E.J. Snell and H. Simpson

A Primer on Linear Models J.F. Monahan

Elements of Simulation B.J.T. Morgan

Analysis of Failure and Survival Data P. J. Smith

Probability: Methods and Measurement A. O’Hagan

Applied Nonparametric Statistical Methods, Fourth Edition P. Sprent and N.C. Smeeton

Applied Bayesian Forecasting and Time Series Analysis A. Pole, M. West, and J. Harrison

Generalized Linear Mixed Models: Modern Concepts, Methods and Applications W. W. Stroup

Introduction to Statistical Limit Theory A.M. Polansky

Data Driven Statistical Methods P. Sprent

Statistics in Research and Development, Time Series: Modeling, Computation, and Inference R. Prado and M. West

Survival Analysis Using S: Analysis of Time-to-Event Data M. Tableman and J.S. Kim

Introduction to Statistical Process Control P. Qiu

Sampling Methodologies with Applications P.S.R.S. Rao A First Course in Linear Model Theory N. Ravishanker and D.K. Dey Essential Statistics, Fourth Edition D.A.G. Rees

Stochastic Modeling and Mathematical Statistics: A Text for Statisticians and Quantitative Scientists F.J. Samaniego

Statistical Methods for Spatial Data Analysis O. Schabenberger and C.A. Gotway Bayesian Networks: With Examples in R M. Scutari and J.-B. Denis Large Sample Methods in Statistics P.K. Sen and J. da Motta Singer

Spatio-Temporal Methods in Environmental Epidemiology G. Shaddick and J.V. Zidek

Applied Categorical and Count Data Analysis W. Tang, H. He, and X.M. Tu

Elementary Applications of Probability Theory, Second Edition H.C. Tuckwell Introduction to Statistical Inference and Its Applications with R M.W. Trosset

Understanding Advanced Statistical Methods P.H. Westfall and K.S.S. Henning Statistical Process Control: Theory and Practice, Third Edition G.B. Wetherill and D.W. Brown Generalized Additive Models: An Introduction with R S. Wood

Epidemiology: Study Design and Data Analysis, Third Edition M. Woodward

Practical Data Analysis for Designed Experiments B.S. Yandell

Texts in Statistical Science

Statistical Rethinking

A Bayesian Course with Examples in R and Stan

Richard McElreath Max Planck Institute for Evolutionary Anthropology Leipzig, Germany

CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2016 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Version Date: 20150910 International Standard Book Number-13: 978-1-4822-5346-7 (eBook - PDF) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http:// www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com

Contents

Preface Audience Teaching strategy How to use this book Installing the rethinking R package Acknowledgments

xi xi xii xii xvi xvi

Chapter 1. The Golem of Prague 1.1. Statistical golems 1.2. Statistical rethinking 1.3. Three tools for golem engineering 1.4. Summary

1 1 4 10 16

Chapter 2. Small Worlds and Large Worlds 2.1. The garden of forking data 2.2. Building a model 2.3. Components of the model 2.4. Making the model go 2.5. Summary 2.6. Practice

19 20 28 32 37 45 45

Chapter 3. Sampling the Imaginary 3.1. Sampling from a grid-approximate posterior 3.2. Sampling to summarize 3.3. Sampling to simulate prediction 3.4. Summary 3.5. Practice

49 52 53 61 68 69

Chapter 4. Linear Models 4.1. Why normal distributions are normal 4.2. A language for describing models 4.3. A Gaussian model of height 4.4. Adding a predictor 4.5. Polynomial regression 4.6. Summary 4.7. Practice

71 72 77 78 92 110 115 115

Chapter 5. Multivariate Linear Models 5.1. Spurious association 5.2. Masked relationship 5.3. When adding variables hurts

119 121 135 141 vii

viii

CONTENTS

5.4. 5.5. 5.6. 5.7.

Categorical variables Ordinary least squares and lm Summary Practice

152 159 162 162

Chapter 6. Overfitting, Regularization, and Information Criteria 6.1. The problem with parameters 6.2. Information theory and model performance 6.3. Regularization 6.4. Information criteria 6.5. Using information criteria 6.6. Summary 6.7. Practice

165 167 174 186 188 195 205 205

Chapter 7. Interactions 7.1. Building an interaction 7.2. Symmetry of the linear interaction 7.3. Continuous interactions 7.4. Interactions in design formulas 7.5. Summary 7.6. Practice

209 211 223 225 235 236 236

Chapter 8. Markov Chain Monte Carlo 8.1. Good King Markov and His island kingdom 8.2. Markov chain Monte Carlo 8.3. Easy HMC: map2stan 8.4. Care and feeding of your Markov chain 8.5. Summary 8.6. Practice

241 242 245 247 255 263 263

Chapter 9. Big Entropy and the Generalized Linear Model 9.1. Maximum entropy 9.2. Generalized linear models 9.3. Maximum entropy priors 9.4. Summary

267 268 280 288 289

Chapter 10. Counting and Classification 10.1. Binomial regression 10.2. Poisson regression 10.3. Other count regressions 10.4. Summary 10.5. Practice

291 292 311 322 328 329

Chapter 11. Monsters and Mixtures 11.1. Ordered categorical outcomes 11.2. Zero-inflated outcomes 11.3. Over-dispersed outcomes 11.4. Summary 11.5. Practice

331 331 342 346 351 352

Chapter 12. Multilevel Models 12.1. Example: Multilevel tadpoles 12.2. Varying effects and the underfitting/overfitting trade-off

355 357 364

CONTENTS

12.3. 12.4. 12.5. 12.6.

More than one type of cluster Multilevel posterior predictions Summary Practice

ix

370 376 384 384

Chapter 13. Adventures in Covariance 13.1. Varying slopes by construction 13.2. Example: Admission decisions and gender 13.3. Example: Cross-classified chimpanzees with varying slopes 13.4. Continuous categories and the Gaussian process 13.5. Summary 13.6. Practice

387 389 398 403 410 419 419

Chapter 14. Missing Data and Other Opportunities 14.1. Measurement error 14.2. Missing data 14.3. Summary 14.4. Practice

423 424 431 439 439

Chapter 15.

441

Horoscopes

Endnotes

445

Bibliography

457

Citation index

465

Topic index

467

This page intentionally left blank

Preface

Masons, when they start upon a building, Are careful to test out the scaffolding; Make sure that planks won’t slip at busy points, Secure all ladders, tighten bolted joints. And yet all this comes down when the job’s done Showing off walls of sure and solid stone. So if, my dear, there sometimes seem to be Old bridges breaking between you and me Never fear. We may let the scaffolds fall Confident that we have built our wall. (“Scaffolding” by Seamus Heaney, 1939–2013) This book means to help you raise your knowledge of and confidence in statistical modeling. It is meant as a scaffold, one that will allow you to construct the wall that you need, even though you will discard it afterwards. As a result, this book teaches the material in often inconvenient fashion, forcing you to perform step-by-step calculations that are usually automated. The reason for all the algorithmic fuss is to ensure that you understand enough of the details to make reasonable choices and interpretations in your own modeling work. So although you will move on to use more automation, it’s important to take things slow at first. Put up your wall, and then let the scaffolding fall.

Audience The principle audience is researchers in the natural and social sciences, whether new PhD students or seasoned professionals, who have had a basic course on regression but nevertheless remain uneasy about statistical modeling. This audience accepts that there is something vaguely wrong about typical statistical practice in the early 21st century, dominated as it is by p-values and a confusing menagerie of testing procedures. They see alternative methods in journals and books. But these people are not sure where to go to learn about these methods. As a consequence, this book doesn’t really argue against p-values and the like. The problem in my opinion isn’t so much p-values as the set of odd rituals that have evolved around xi

xii

PREFACE

them, in the wilds of the sciences, as well as the exclusion of so many other useful tools. So the book assumes the reader is ready to try doing statistical inference without p-values. This isn’t the ideal situation. It would be better to have material that helps you spot common mistakes and misunderstandings of p-values and tests in general, as all of us have to understand such things, even if we don’t use them. So I’ve tried to sneak in a little material of that kind, but unfortunately cannot devote much space to it. The book would be too long, and it would disrupt the teaching flow of the material. It’s important to realize, however, that the disregard paid to p-values is not a uniquely Bayesian attitude. Indeed, significance testing can be—and has been—formulated as a Bayesian procedure as well. So the choice to avoid significance testing is stimulated instead by epistemological concerns, some of which are briefly discussed in the first chapter.

Teaching strategy The book uses much more computer code than formal mathematics. Even excellent mathematicians can have trouble understanding an approach, until they see a working algorithm. This is because implementation in code form removes all ambiguities. So material of this sort is easier to learn, if you also learn how to implement it. In addition to any pedagogical value of presenting code, so much of statistics is now computational that a purely mathematical approach is anyways insufficient. As you’ll see in later parts of this book, the same mathematical statistical model can sometimes be implemented in different ways, and the differences matter. So when you move beyond this book to more advanced or specialized statistical modeling, the computational emphasis here will help you recognize and cope with all manner of practical troubles. Every section of the book is really just the tip of an iceberg. I’ve made no attempt to be exhaustive. Rather I’ve tried to explain something well. In this attempt, I’ve woven a lot of concepts and material into data analysis examples. So instead of having traditional units on, for example, centering predictor variables, I’ve developed those concepts in the context of a narrative about data analysis. This is certainly not a style that works for all readers. But it has worked for a lot of my students. I suspect it fails dramatically for those who are being forced to learn this information. For the internally motivated, it reflects how we really learn these skills in the context of our research.

How to use this book This book is not a reference, but a course. It doesn’t try to support random access. Rather, it expects sequential access. This has immense pedagogical advantages, but it has the disadvantage of violating how most scientists actually read books. This book has a lot of code in it, integrated fully into the main text. The reason for this is that doing model-based statistics in the 21st century really requires programming, of at least a minor sort. The code is not optional. Everyplace, I have erred on the side of including too much code, rather than too little. In my experience teaching scientific programming, novices learn more quickly when they have working code to modify, rather than needing to write an algorithm from scratch. My generation was probably the last to have to learn some programming to use a computer, and so coding has gotten harder and harder to teach as time goes on. My students are very computer literate, but they have no idea what computer code looks like.

HOW TO USE THIS BOOK

xiii

What the book assumes. This book does not try to teach the reader to program, in the most basic sense. It assumes that you have made a basic effort to learn how to install and process data in R. In most cases, a short introduction to R programming will be enough. I know many people have found Emmanuel Paradis’ R for Beginners helpful. You can find it and many other beginner guides here: http://cran.r-project.org/other-docs.html To make use of this book, you should know already that y