Hansen. Econometrics

Citation preview

ECONOMETRICS Bruce E. Hansen c 2000, 20121

University of Wisconsin

www.ssc.wisc.edu/~bhansen This Revision: January 18, 2012 Comments Welcome

1

This manuscript may be printed and reproduced for individual or instructional use, but may not be printed for commercial purposes.

Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii 1 Introduction 1.1 What is Econometrics? . . . . . . . . . . . . 1.2 The Probability Approach to Econometrics 1.3 Econometric Terms and Notation . . . . . . 1.4 Observational Data . . . . . . . . . . . . . . 1.5 Standard Data Structures . . . . . . . . . . 1.6 Sources for Economic Data . . . . . . . . . 1.7 Econometric Software . . . . . . . . . . . . 1.8 Reading the Manuscript . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

1 1 1 2 3 4 5 6 7

2 Moment Estimation 2.1 Introduction . . . . . . . . . . 2.2 Population and Sample Mean 2.3 Sample Mean is Unbiased . . 2.4 Variance . . . . . . . . . . . . 2.5 Convergence in Probability . 2.6 Weak Law of Large Numbers 2.7 Vector-Valued Moments . . . 2.8 Convergence in Distribution . 2.9 Functions of Moments . . . . 2.10 Delta Method . . . . . . . . . 2.11 Stochastic Order Symbols . . 2.12 Uniform Stochastic Bounds* . 2.13 Semiparametric E¢ciency . . 2.14 Expectation* . . . . . . . . . 2.15 Technical Proofs* . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

8 8 8 9 9 10 11 12 14 15 17 18 20 21 24 25

3 Conditional Expectation and Projection 3.1 Introduction . . . . . . . . . . . . . . . . 3.2 The Distribution of Wages . . . . . . . . 3.3 Conditional Expectation . . . . . . . . . 3.4 Conditional Expectation Function . . . 3.5 Continuous Variables . . . . . . . . . . . 3.6 Law of Iterated Expectations . . . . . . 3.7 Monotonicity of Conditioning . . . . . . 3.8 CEF Error . . . . . . . . . . . . . . . . . 3.9 Best Predictor . . . . . . . . . . . . . . 3.10 Conditional Variance . . . . . . . . . . . 3.11 Homoskedasticity and Heteroskedasticity 3.12 Regression Derivative . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

29 29 29 31 34 36 37 38 39 41 41 43 43

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

i

ii

CONTENTS 3.13 Linear CEF . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.14 Linear CEF with Nonlinear E¤ects . . . . . . . . . . . . . . 3.15 Linear CEF with Dummy Variables . . . . . . . . . . . . . . 3.16 Best Linear Predictor . . . . . . . . . . . . . . . . . . . . . 3.17 Linear Predictor Error Variance . . . . . . . . . . . . . . . . 3.18 Regression Coe¢cients . . . . . . . . . . . . . . . . . . . . . 3.19 Regression Sub-Vectors . . . . . . . . . . . . . . . . . . . . 3.20 Coe¢cient Decomposition . . . . . . . . . . . . . . . . . . . 3.21 Omitted Variable Bias . . . . . . . . . . . . . . . . . . . . . 3.22 Best Linear Approximation . . . . . . . . . . . . . . . . . . 3.23 Normal Regression . . . . . . . . . . . . . . . . . . . . . . . 3.24 Regression to the Mean . . . . . . . . . . . . . . . . . . . . 3.25 Reverse Regression . . . . . . . . . . . . . . . . . . . . . . . 3.26 Limitations of the Best Linear Predictor . . . . . . . . . . . 3.27 Random Coe¢cient Model . . . . . . . . . . . . . . . . . . . 3.28 Causal E¤ects . . . . . . . . . . . . . . . . . . . . . . . . . . 3.29 Existence and Uniqueness of the Conditional Expectation* 3.30 Technical Proofs* . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 The Algebra of Least Squares 4.1 Introduction . . . . . . . . . . 4.2 Least Squares Estimator . . . 4.3 Solving for Least Squares . . 4.4 Illustration . . . . . . . . . . 4.5 Least Squares Residuals . . . 4.6 Model in Matrix Notation . . 4.7 Projection Matrix . . . . . . 4.8 Orthogonal Projection . . . . 4.9 Analysis of Variance . . . . . 4.10 Regression Components . . . 4.11 Residual Regression . . . . . 4.12 Prediction Errors . . . . . . . 4.13 In‡uential Observations . . . 4.14 Normal Regression Model . . 4.15 Technical Proofs* . . . . . . . Exercises . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

5 Least Squares Regression 5.1 Introduction . . . . . . . . . . . . . . 5.2 Mean of Least-Squares Estimator . . 5.3 Variance of Least Squares Estimator 5.4 Gauss-Markov Theorem . . . . . . . 5.5 Residuals . . . . . . . . . . . . . . . 5.6 Estimation of Error Variance . . . . 5.7 Mean-Square Forecast Error . . . . . 5.8 Covariance Matrix Estimation Under 5.9 Covariance Matrix Estimation Under 5.10 Standard Errors . . . . . . . . . . . . 5.11 Measures of Fit . . . . . . . . . . . . 5.12 Empirical Example . . . . . . . . . . 5.13 Multicollinearity . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Homoskedasticity Heteroskedasticity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

44 45 46 48 54 55 55 56 57 58 58 59 60 61 62 63 66 67 70

. . . . . . . . . . . . . . . .

72 72 72 73 75 75 76 78 79 80 81 82 84 84 86 87 89

. . . . . . . . . . . . .

92 92 93 94 95 97 98 99 101 101 104 105 106 107

iii

CONTENTS

5.14 Normal Regression Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 6 Asymptotic Theory for Least Squares 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Consistency of Least-Squares Estimation . . . . . . . . . . . . . . . . 6.3 Asymptotic Normality . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Joint Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Consistency of Error Variance Estimators . . . . . . . . . . . . . . . 6.6 Homoskedastic Covariance Matrix Estimation . . . . . . . . . . . . . 6.7 Asymptotic Covariance Matrix Estimation . . . . . . . . . . . . . . . 6.8 Alternative Covariance Matrix Estimation* . . . . . . . . . . . . . . 6.9 Functions of Parameters . . . . . . . . . . . . . . . . . . . . . . . . . 6.10 Asymptotic Standard Errors . . . . . . . . . . . . . . . . . . . . . . . 6.11 t statistic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.12 Con…dence Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.13 Regression Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.14 Forecast Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.15 Wald Statistic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.16 Con…dence Regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.17 Semiparametric E¢ciency in the Projection Model . . . . . . . . . . 6.18 Semiparametric E¢ciency in the Homoskedastic Regression Model* . 6.19 Uniformly Consistent Residuals* . . . . . . . . . . . . . . . . . . . . 6.20 Asymptotic Leverage* . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

113 113 113 116 118 122 123 123 125 126 128 128 129 130 133 133 134 136 137 138 139 141

7 Restricted Estimation 7.1 Introduction . . . . . . . . . . . . . . . . 7.2 Constrained Least Squares . . . . . . . . 7.3 Exclusion Restriction . . . . . . . . . . . 7.4 Minimum Distance . . . . . . . . . . . . 7.5 Asymptotic Distribution . . . . . . . . . 7.6 E¢cient Minimum Distance Estimator . 7.7 Exclusion Restriction Revisited . . . . . 7.8 Variance and Standard Error Estimation 7.9 Misspeci…cation . . . . . . . . . . . . . . 7.10 Nonlinear Constraints . . . . . . . . . . 7.11 Inequality Constraints . . . . . . . . . . 7.12 Constrained MLE . . . . . . . . . . . . . 7.13 Technical Proofs* . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

143 143 144 145 145 146 147 148 150 150 152 153 153 154 156

. . . . . . . . .

157 . 157 . 160 . 162 . 163 . 164 . 165 . 166 . 167 . 170

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

8 Hypothesis Testing 8.1 Hypotheses and Tests . . . . . . . . . . . . . . 8.2 t tests . . . . . . . . . . . . . . . . . . . . . . . 8.3 t-ratios and the Abuse of Testing . . . . . . . . 8.4 Wald Tests . . . . . . . . . . . . . . . . . . . . 8.5 Minimum Distance Tests . . . . . . . . . . . . . 8.6 F Tests . . . . . . . . . . . . . . . . . . . . . . 8.7 Likelihood Ratio Test . . . . . . . . . . . . . . 8.8 Problems with Tests of NonLinear Hypotheses 8.9 Monte Carlo Simulation . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

iv

CONTENTS

8.10 Con…dence Intervals by Test Inversion . . . . . . . . . . . . . . . . . . . . . . . . . . 172 8.11 Asymptotic Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 9 Regression Extensions 9.1 Generalized Least Squares . . . . . 9.2 Testing for Heteroskedasticity . . . 9.3 NonLinear Least Squares . . . . . 9.4 Testing for Omitted NonLinearity . Exercises . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

178 178 181 181 183 185

10 The Bootstrap 10.1 De…nition of the Bootstrap . . . . . . . . . 10.2 The Empirical Distribution Function . . . . 10.3 Nonparametric Bootstrap . . . . . . . . . . 10.4 Bootstrap Estimation of Bias and Variance 10.5 Percentile Intervals . . . . . . . . . . . . . . 10.6 Percentile-t Equal-Tailed Interval . . . . . . 10.7 Symmetric Percentile-t Intervals . . . . . . 10.8 Asymptotic Expansions . . . . . . . . . . . 10.9 One-Sided Tests . . . . . . . . . . . . . . . 10.10Symmetric Two-Sided Tests . . . . . . . . . 10.11Percentile Con…dence Intervals . . . . . . . 10.12Bootstrap Methods for Regression Models . Exercises . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

186 186 186 188 188 189 191 191 192 194 195 196 197 199

. . . . . . . . . .

200 . 200 . 200 . 201 . 203 . 205 . 206 . 209 . 211 . 212 . 213

. . . . . . . . . . . . .

215 . 215 . 215 . 217 . 217 . 217 . 218 . 219 . 220 . 220 . 221 . 221 . 223 . 224

. . . . .

. . . . .

. . . . .

11 NonParametric Regression 11.1 Introduction . . . . . . . . . . . . . . . . 11.2 Binned Estimator . . . . . . . . . . . . . 11.3 Kernel Regression . . . . . . . . . . . . . 11.4 Local Linear Estimator . . . . . . . . . . 11.5 Nonparametric Residuals and Regression 11.6 Cross-Validation Bandwidth Selection . 11.7 Asymptotic Distribution . . . . . . . . . 11.8 Conditional Variance Estimation . . . . 11.9 Standard Errors . . . . . . . . . . . . . . 11.10Multiple Regressors . . . . . . . . . . . . 12 Series Estimation 12.1 Approximation by Series . . . . . 12.2 Splines . . . . . . . . . . . . . . . 12.3 Partially Linear Model . . . . . . 12.4 Additively Separable Models . . 12.5 Uniform Approximation . . . . . 12.6 Runge’s Phenonmenon . . . . . . 12.7 Approximating Regression . . . . 12.8 Estimation . . . . . . . . . . . . 12.9 Residuals and Regression Fit . . 12.10Cross-Validation Model Selection 12.11Convergence Rates . . . . . . . . 12.12Uniform Convergence . . . . . . . 12.13Asymptotic Normality . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . .

. . . . . . . . . . . . Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

v

CONTENTS

12.14Regression Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 12.15Kernel Versus Series Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 12.16Technical Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 13 Quantile Regression 231 13.1 Least Absolute Deviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 13.2 Quantile Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 14 Generalized Method of Moments 14.1 Overidenti…ed Linear Model . . . . . . . . . 14.2 GMM Estimator . . . . . . . . . . . . . . . 14.3 Distribution of GMM Estimator . . . . . . 14.4 Estimation of the E¢cient Weight Matrix . 14.5 GMM: The General Case . . . . . . . . . . 14.6 Over-Identi…cation Test . . . . . . . . . . . 14.7 Hypothesis Testing: The Distance Statistic 14.8 Conditional Moment Restrictions . . . . . . 14.9 Bootstrap GMM Inference . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . 15 Empirical Likelihood 15.1 Non-Parametric Likelihood . . . . . . . . 15.2 Asymptotic Distribution of EL Estimator 15.3 Overidentifying Restrictions . . . . . . . . 15.4 Testing . . . . . . . . . . . . . . . . . . . . 15.5 Numerical Computation . . . . . . . . . . 16 Endogeneity 16.1 Instrumental Variables . . . 16.2 Reduced Form . . . . . . . 16.3 Identi…cation . . . . . . . . 16.4 Estimation . . . . . . . . . 16.5 Special Cases: IV and 2SLS 16.6 Bekker Asymptotics . . . . 16.7 Identi…cation Failure . . . . Exercises . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

17 Univariate Time Series 17.1 Stationarity and Ergodicity . . . . . . 17.2 Autoregressions . . . . . . . . . . . . . 17.3 Stationarity of AR(1) Process . . . . . 17.4 Lag Operator . . . . . . . . . . . . . . 17.5 Stationarity of AR(k) . . . . . . . . . 17.6 Estimation . . . . . . . . . . . . . . . 17.7 Asymptotic Distribution . . . . . . . . 17.8 Bootstrap for Autoregressions . . . . . 17.9 Trend Stationarity . . . . . . . . . . . 17.10Testing for Omitted Serial Correlation 17.11Model Selection . . . . . . . . . . . . . 17.12Autoregressive Unit Roots . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

237 . 237 . 238 . 239 . 240 . 241 . 241 . 242 . 243 . 244 . 246

. . . . .

248 . 248 . 250 . 251 . 252 . 253

. . . . . . . .

255 . 256 . 257 . 258 . 258 . 258 . 260 . 261 . 263

. . . . . . . . . . . .

265 . 265 . 267 . 268 . 268 . 269 . 269 . 270 . 271 . 271 . 272 . 273 . 273

vi

CONTENTS 18 Multivariate Time Series 18.1 Vector Autoregressions (VARs) . . . . 18.2 Estimation . . . . . . . . . . . . . . . 18.3 Restricted VARs . . . . . . . . . . . . 18.4 Single Equation from a VAR . . . . . 18.5 Testing for Omitted Serial Correlation 18.6 Selection of Lag Length in an VAR . . 18.7 Granger Causality . . . . . . . . . . . 18.8 Cointegration . . . . . . . . . . . . . . 18.9 Cointegrated VARs . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

275 275 276 276 276 277 277 278 278 279

19 Limited Dependent Variables 19.1 Binary Choice . . . . . . . . 19.2 Count Data . . . . . . . . . 19.3 Censored Data . . . . . . . 19.4 Sample Selection . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

281 281 282 283 284

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

20 Panel Data 286 20.1 Individual-E¤ects Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 20.2 Fixed E¤ects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 20.3 Dynamic Panel Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 21 Nonparametric Density Estimation 289 21.1 Kernel Density Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 21.2 Asymptotic MSE for Kernel Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . 291 A Matrix Algebra A.1 Notation . . . . . . . . . . . A.2 Matrix Addition . . . . . . A.3 Matrix Multiplication . . . A.4 Trace . . . . . . . . . . . . . A.5 Rank and Inverse . . . . . . A.6 Determinant . . . . . . . . . A.7 Eigenvalues . . . . . . . . . A.8 Positive De…niteness . . . . A.9 Matrix Calculus . . . . . . . A.10 Kronecker Products and the A.11 Vector and Matrix Norms . A.12 Matrix Inequalities . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

294 . 294 . 295 . 295 . 296 . 297 . 298 . 299 . 300 . 301 . 301 . 302 . 302

B Probability B.1 Foundations . . . . . . . . . . . . . . . . . . B.2 Random Variables . . . . . . . . . . . . . . B.3 Expectation . . . . . . . . . . . . . . . . . . B.4 Gamma Function . . . . . . . . . . . . . . . B.5 Common Distributions . . . . . . . . . . . . B.6 Multivariate Random Variables . . . . . . . B.7 Conditional Distributions and Expectation . B.8 Transformations . . . . . . . . . . . . . . . B.9 Normal and Related Distributions . . . . . B.10 Inequalities . . . . . . . . . . . . . . . . . . B.11 Maximum Likelihood . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vec Operator . . . . . . . . . . . . . . . .

306 306 308 308 309 310 312 314 316 317 319 322

CONTENTS C Numerical Optimization C.1 Grid Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.2 Gradient Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.3 Derivative-Free Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii 327 . 327 . 327 . 329

Preface This book is intended to serve as the textbook for a …rst-year graduate course in econometrics. It can be used as a stand-alone text, or be used as a supplement to another text. Students are assumed to have an understanding of multivariate calculus, probability theory, linear algebra, and mathematical statistics. A prior course in undergraduate econometrics would be helpful, but not required. For reference, some of the basic tools of matrix algebra, probability, and statistics are reviewed in the Appendix. For students wishing to deepen their knowledge of matrix algebra in relation to their study of econometrics, I recommend Matrix Algebra by Abadir and Magnus (2005). An excellent introduction to probability and statistics is Statistical Inference by Casella and Berger (2002). For those wanting a deeper foundation in probability, I recommend Ash (1972) or Billingsley (1995). For more advanced statistical theory, I recommend Lehmann and Casella (1998), van der Vaart (1998), Shao (2003), and Lehmann and Romano (2005). For further study in econometrics beyond this text, I recommend Davidson (1994) for asymptotic theory, Hamilton (1994) for time-series methods, Wooldridge (2002) for panel data and discrete response models, and Li and Racine (2007) for nonparametrics and semiparametric econometrics. Beyond these texts, the Handbook of Econometrics series provides advanced summaries of contemporary econometric methods and theory. The end-of-chapter exercises are important parts of the text and are meant to help teach students of econometrics. Answers are not provided, and this is intentional. I would like to thank Ying-Ying Lee for providing research assistance in preparing some of the empirical examples presented in the text. As this is a manuscript in progress, some parts are quite incomplete, and there are many topics which I plan to add. In general chapters 1-8 are the most complete, the remaining need signi…cant work and revision.

viii

Chapter 1

Introduction 1.1

What is Econometrics?

The term “econometrics” is believed to have been crafted by Ragnar Frisch (1895-1973) of Norway, one of the three principle founders of the Econometric Society, …rst editor of the journal Econometrica, and co-winner of the …rst Nobel Memorial Prize in Economic Sciences in 1969. It is therefore …tting that we turn to Frisch’s own words in the introduction to the …rst issue of Econometrica for an explanation of the discipline. A word of explanation regarding the term econometrics may be in order. Its de…nition is implied in the statement of the scope of the [Econometric] Society, in Section I of the Constitution, which reads: “The Econometric Society is an international society for the advancement of economic theory in its relation to statistics and mathematics.... Its main object shall be to promote studies that aim at a uni…cation of the theoreticalquantitative and the empirical-quantitative approach to economic problems....” But there are several aspects of the quantitative approach to economics, and no single one of these aspects, taken by itself, should be confounded with econometrics. Thus, econometrics is by no means the same as economic statistics. Nor is it identical with what we call general economic theory, although a considerable portion of this theory has a de…ninitely quantitative character. Nor should econometrics be taken as synonomous with the application of mathematics to economics. Experience has shown that each of these three view-points, that of statistics, economic theory, and mathematics, is a necessary, but not by itself a su¢cient, condition for a real understanding of the quantitative relations in modern economic life. It is the uni…cation of all three that is powerful. And it is this uni…cation that constitutes econometrics. Ragnar Frisch, Econometrica, (1933), 1, pp. 1-2. This de…nition remains valid today, although some terms have evolved somewhat in their usage. Today, we would say that econometrics is the uni…ed study of economic models, mathematical statistics, and economic data. Within the …eld of econometrics there are sub-divisions and specializations. Econometric theory concerns the development of tools and methods, and the study of the properties of econometric methods. Applied econometrics is a term describing the development of quantitative economic models and the application of econometric methods to these models using economic data.

1.2

The Probability Approach to Econometrics

The unifying methodology of modern econometrics was articulated by Trygve Haavelmo (19111999) of Norway, winner of the 1989 Nobel Memorial Prize in Economic Sciences, in his seminal 1

CHAPTER 1. INTRODUCTION

2

paper “The probability approach in econometrics”, Econometrica (1944). Haavelmo argued that quantitative economic models must necessarily be probability models (by which today we would mean stochastic). Deterministic models are blatently inconsistent with observed economic quantities, and it is incohorent to apply deterministic models to non-deterministic data. Economic models should be explicitly designed to incorporate randomness; stochastic errors should not be simply added to deterministic models to make them random. Once we acknowledge that an economic model is a probability model, it follows naturally that the best way to quantify, estimate, and conduct inferences about the economy is through the powerful theory of mathematical statistics. The appropriate method for a quantitative economic analysis follows from the probabilistic construction of the economic model. Haavelmo’s probability approach was quickly embraced by the economics profession. Today no quantitative work in economics shuns its fundamental vision. While all economists embrace the probability approach, there has been some evolution in its implementation. The structural approach is the closest to Haavelmo’s original idea. A probabilistic economic model is speci…ed, and the quantitative analysis performed under the assumption that the economic model is correctly speci…ed. Researchers often describe this as “taking their model seriously.” The structural approach typically leads to likelihood-based analysis, including maximum likelihood and Bayesian estimation. A criticism of the structural approach is that it is misleading to treat an economic model as correctly speci…ed. Rather, it is more accurate to view a model as a useful abstraction or approximation. In this case, how should we interpret structural econometric analysis? The quasistructural approach to inference views a structural economic model as an approximation rather than the truth. This theory has led to the concepts of the pseudo-true value (the parameter value de…ned by the estimation problem), the quasi-likelihood function, quasi-MLE, and quasi-likelihood inference. Closely related is the semiparametric approach. A probabilistic economic model is partially speci…ed but some features are left unspeci…ed. This approach typically leads to estimation methods such as least-squares and the Generalized Method of Moments. The semiparametric approach dominates contemporary econometrics, and is the main focus of this textbook. Another branch of quantitative structural economics is the calibration approach. Similar to the quasi-structural approach, the calibration approach interprets structural models as approximations and hence inherently false. The di¤erence is that the calibrationist literature rejects mathematical statistics as inappropriate for approximate models, and instead selects parameters by matching model and data moments using non-statistical ad hoc 1 methods.

1.3

Econometric Terms and Notation

In a typical application, an econometrician has a set of repeated measurements on a set of variables. For example, in a labor application the variables could include weekly earnings, educational attainment, age, and other descriptive characteristics. We call this information the data, dataset, or sample. We use the term observations to refer to the distinct repeated measurements on the variables. An individual observation often corresponds to a speci…c economic unit, such as a person, household, corporation, …rm, organization, country, state, city or other geographical region. An individual observation could also be a measurement at a point in time, such as quarterly GDP or a daily interest rate. Economists typically denote variables by the italicized roman characters y, x; and/or z: The convention in econometrics is to use the character y to denote the variable to be explained, while 1 Ad hoc means “for this purpose” – a method designed for a speci…c problem – and not based on a generalizable principle.

3

CHAPTER 1. INTRODUCTION

the characters x and z are used to denote the conditioning (explaining) variables. Following mathematical convention, real numbers (elements of the real line R) are written using lower case italics such as y, and vectors (elements of Rk ) by lower case bold italics such as x; e.g. 1 0 x1 B x2 C C B x = B . C: @ .. A xk

Upper case bold italics such as X are used for matrices. We typically denote the number of observations by the natural number n; and subscript the variables by the index i to denote the individual observation, e.g. yi ; xi and z i . In some contexts we use indices other than i, such as in time-series applications where the index t is common, and in panel studies we typically use the double index it to refer to individual i at a time period t.

The i’th observation is the set (yi ; xi ; z i ):

It is proper mathematical practice to use upper case X for random variables and lower case x for realizations or speci…c values. This practice is not commonly followed in econometrics because instead we use upper case to denote matrices. Thus the notation yi will in some places refer to a random variable, and in other places a speci…c realization. Hopefully there will be no confusion as the use should be evident from the context. We typically use Greek letters such as ; and 2 to denote unknown parameters of an econometric model, and will use boldface, e.g. or , when these are vector-valued. Estimates are typically denoted by putting a hat “^”, tilde “~” or bar “-” over the corresponding letter, e.g. ^ and ~ are estimates of : The covariance matrix of an econometric estimator will typically be written using the capital p b boldface V ; often with a subscript to denote the estimator, e.g. V b = var as the n p b : Hopefully without causing confusion, we will use the notation covariance matrix for n p V = avar( b ) to denote the asymptotic covariance matrix of n b (the variance of the asymptotic distribution). Estimates will be denoted by appending hats or tildes, e.g. Vb is an estimate of V .

1.4

Observational Data

A common econometric question is to quantify the impact of one set of variables on another variable. For example, a concern in labor economics is the returns to schooling – the change in earnings induced by increasing a worker’s education, holding other variables constant. Another issue of interest is the earnings gap between men and women. Ideally, we would use experimental data to answer these questions. To measure the returns to schooling, an experiment might randomly divide children into groups, mandate di¤erent levels of education to the di¤erent groups, and then follow the children’s wage path after they mature and enter the labor force. The di¤erences between the groups would be direct measurements of the effects of di¤erent levels of education. However, experiments such as this would be widely condemned as immoral! Consequently, we see few non-laboratory experimental data sets in economics. Instead, most economic data is observational. To continue the above example, through data collection we can record the level of a person’s education and their wage. With such data we can measure the joint distribution of these variables, and assess the joint dependence. But from

4

CHAPTER 1. INTRODUCTION

observational data it is di¢cult to infer causality, as we are not able to manipulate one variable to see the direct e¤ect on the other. For example, a person’s level of education is (at least partially) determined by that person’s choices. These factors are likely to be a¤ected by their personal abilities and attitudes towards work. The fact that a person is highly educated suggests a high level of ability, which suggests a high relative wage. This is an alternative explanation for an observed positive correlation between educational levels and wages. High ability individuals do better in school, and therefore choose to attain higher levels of education, and their high ability is the fundamental reason for their high wages. The point is that multiple explanations are consistent with a positive correlation between schooling levels and education. Knowledge of the joint distibution alone may not be able to distinguish between these explanations.

Most economic data sets are observational, not experimental. This means that all variables must be treated as random and possibly jointly determined.

This discussion means that it is di¢cult to infer causality from observational data alone. Causal inference requires identi…cation, and this is based on strong assumptions. We will return to a discussion of some of these issues in Chapter 16.

1.5

Standard Data Structures

There are three major types of economic data sets: cross-sectional, time-series, and panel. They are distinguished by the dependence structure across observations. Cross-sectional data sets have one observation per individual. Surveys are a typical source for cross-sectional data. In typical applications, the individuals surveyed are persons, households, …rms or other economic agents. In many contemporary econometric cross-section studies the sample size n is quite large. It is conventional to assume that cross-sectional observations are mutually independent. Most of this text is devoted to the study of cross-section data. Time-series data are indexed by time. Typical examples include macroeconomic aggregates, prices and interest rates. This type of data is characterized by serial dependence so the random sampling assumption is inappropriate. Most aggregate economic data is only available at a low frequency (annual, quarterly or perhaps monthly) so the sample size is typically much smaller than in cross-section studies. The exception is …nancial data where data are available at a high frequency (weekly, daily, hourly, or by transaction) so sample sizes can be quite large. Panel data combines elements of cross-section and time-series. These data sets consist of a set of individuals (typically persons, households, or corporations) surveyed repeatedly over time. The common modeling assumption is that the individuals are mutually independent of one another, but a given individual’s observations are mutually dependent. This is a modi…ed random sampling environment.

Data Structures Cross-section Time-series Panel

CHAPTER 1. INTRODUCTION

5

Some contemporary econometric applications combine elements of cross-section, time-series, and panel data modeling. These include models of spatial correlation and clustering. As we mentioned above, most of this text will be devoted to cross-sectional data under the assumption of mutually independent observations. By mutual independence we mean that the i’th observation (yi ; xi ; z i ) is independent of the j’th observation (yj ; xj ; z j ) for i 6= j. (Sometimes the label “independent” is misconstrued. It is a statement about the relationship between observations i and j, not a statement about the relationship between yi and xi and/or z i :) Furthermore, if the data is randomly gathered, it is reasonable to model each observation as a random draw from the same probability distribution. In this case we say that the data are independent and identically distributed or iid. We call this a random sample. For most of this text we will assume that our observations come from a random sample.

De…nition 1.5.1 The observations (yi ; xi ; z i ) are a random sample if they are mutually independent and identically distributed (iid) across i = 1; :::; n:

In the random sampling framework, we think of an individual observation (yi ; xi ; z i ) as a realization from a joint probability distribution F (y; x; z) which can call the population. This “population” is in…nitely large. This abstraction can be a source of confusion as it does not correspond to a physical population in the real world. The distribution F is unknown, and the goal of statistical inference is to learn about features of F from the sample. The assumption of random sampling provides the mathematical foundation for treating economic statistics with the tools of mathematical statistics. The random sampling framework was a major intellectural breakthrough of the late 19th century, allowing the application of mathematical statistics to the social sciences. Before this conceptual development, methods from mathematical statistics had not been applied to economic data as they were viewed as inappropriate. The random sampling framework enabled economic samples to be viewed as homogenous and random, a necessary precondition for the application of statistical methods.

1.6

Sources for Economic Data

Fortunately for economists, the internet provides a convenient forum for dissemination of economic data. Many large-scale economic datasets are available without charge from governmental agencies. An excellent starting point is the Resources for Economists Data Links, available at rfe.org. From this site you can …nd almost every publically available economic data set. Some speci…c data sources of interest include Bureau of Labor Statistics US Census Current Population Survey Survey of Income and Program Participation Panel Study of Income Dynamics Federal Reserve System (Board of Governors and regional banks) National Bureau of Economic Research

CHAPTER 1. INTRODUCTION

6

U.S. Bureau of Economic Analysis CompuStat International Financial Statistics Another good source of data is from authors of published empirical studies. Most journals in economics require authors of published papers to make their datasets generally available. For example, in its instructions for submission, Econometrica states: Econometrica has the policy that all empirical, experimental and simulation results must be replicable. Therefore, authors of accepted papers must submit data sets, programs, and information on empirical analysis, experiments and simulations that are needed for replication and some limited sensitivity analysis. The American Economic Review states: All data used in analysis must be made available to any researcher for purposes of replication. The Journal of Political Economy states: It is the policy of the Journal of Political Economy to publish papers only if the data used in the analysis are clearly and precisely documented and are readily available to any researcher for purposes of replication. If you are interested in using the data from a published paper, …rst check the journal’s website, as many journals archive data and replication programs online. Second, check the website(s) of the paper’s author(s). Most academic economists maintain webpages, and some make available replication …les complete with data and programs. If these investigations fail, email the author(s), politely requesting the data. You may need to be persistent. As a matter of professional etiquette, all authors absolutely have the obligation to make their data and programs available. Unfortunately, many fail to do so, and typically for poor reasons. The irony of the situation is that it is typically in the best interests of a scholar to make as much of their work (including all data and programs) freely available, as this only increases the likelihood of their work being cited and having an impact. Keep this in mind as you start your own empirical project. Remember that as part of your end product, you will need (and want) to provide all data and programs to the community of scholars. The greatest form of ‡attery is to learn that another scholar has read your paper, wants to extend your work, or wants to use your empirical methods. In addition, public openness provides a healthy incentive for transparency and integrity in empirical analysis.

1.7

Econometric Software

Economists use a variety of econometric, statistical, and programming software. STATA (www.stata.com) is a powerful statistical program with a broad set of pre-programmed econometric and statistical tools. It is quite popular among economists, and is continuously being updated with new methods. It is an excellent package for most econometric analysis, but is limited when you want to use new or less-common econometric methods which have not yet been programed. GAUSS (www.aptech.com), MATLAB (www.mathworks.com), and Ox (www.oxmetrics.net) are high-level matrix programming languages with a wide variety of built-in statistical functions. Many econometric methods have been programed in these languages and are available on the web. The advantage of these packages is that you are in complete control of your analysis, and it is

CHAPTER 1. INTRODUCTION

7

easier to program new methods than in STATA. Some disadvantages are that you have to do much of the programming yourself, programming complicated procedures takes signi…cant time, and programming errors are hard to prevent and di¢cult to detect and eliminate. R (www.r-project.org) is an integrated suite of statistical and graphical software that is ‡exible, open source, and best of all, free! For highly-intensive computational tasks, some economists write their programs in a standard programming language such as Fortran or C. This can lead to major gains in computational speed, at the cost of increased time in programming and debugging. As these di¤erent packages have distinct advantages, many empirical economists end up using more than one package. As a student of econometrics, you will learn at least one of these packages, and probably more than one.

1.8

Reading the Manuscript

Chapter 2 is a review of moment estimation and asymptotic distribution theory. This material should be familiar from an earlier course in statistics, but I have included this at the beginning because of its central importance in econometric distribution theory. Chapters 3 through 9 deal with the core linear regression and projection models. Chapter 10 introduces the bootstrap. Chapters 11 through 13 deal with the Generalized Method of Moments, empirical likelihood and endogeneity. Chapters 14 and 15 cover time series, and Chapters 16, 17 and 18 cover limited dependent variables, panel data, and nonparametrics. Reviews of matrix algebra, probability theory, maximum likelihood, and numerical optimization can be found in the appendix. Technical sections which may not be of interest to all readers are marked with an asterisk (*).

Chapter 2

Moment Estimation 2.1

Introduction

Most econometric estimators can be written as functions of sample moments. To understand econometric estimation we need a thorough understanding of moment estimation. This chapter provides a concise summary. It will be useful for most students to review this material, even if most is familiar.

2.2

Population and Sample Mean

A random variable y with density f has the expectation or mean1 = E (y) Z 1 = uf (u)du: 1

This is the average value of y in the population. We would like to estimate from a random sample. Recall that a random sample fy1 ; :::; yn g consists of n observations of independent and identically draws from the distribution of y:

Assumption 2.2.1 The observations fy1 ; :::; yn g are a random sample.

As is the average value of y in the population, it seems reasonable to estimate average value of y in the sample. This is the sample mean, written as

from the

n

1X yn ) = yi n

1 y = (y1 + n

i=1

It is important to understand the distinction between and y. The population mean is a nonrandom feature of the population while the sample mean y is a random feature of a random sample. is …xed, while y varies with the sample. We use the term “mean” to refer to both, but they are really quite distinct. Here, as is common in econometrics, we put a bar “ ” over y to indicate that the quantity is a sample mean. This convention is useful as it helps readers recognize a sample mean. It is also common to see the notation y n ; where the subscript “n” indicates that the sample mean depends on the sample size n: 1

For a rigorous treatment of expectation see Section 2.14.

8

9

CHAPTER 2. MOMENT ESTIMATION

Moment estimation uses sample moments as estimates of population moments. In the case of the mean, the moment estimate of the population mean = Ey is the sample mean b = y: Here, as is common in econometrics, we put a hat “^” over the parameter to indicate that b is a sample estimate of : This is a helpful convention, as just by seeing the symbol b we understand that it is a sample estimate of a population parameter :

2.3

Sample Mean is Unbiased

Since the sample mean is a linear function of the observations, it is simple to calculate its expectation. ! n n 1X 1X Ey = E yi = Eyi = : n n i=1

i=1

This shows that the expected value of the sample mean equals the population mean. An estimator with this property is called unbiased. De…nition 2.3.1 An estimator b for

is unbiased if Eb = .

Theorem 2.3.1 If E jyj < 1 then Ey = population mean :

and b = y is unbiased for the

You may notice that we slipped in the additional condition “If E jyj < 1”. This assumption ensures that is …nite and the mean of y is well de…ned.

2.4

Variance

The variance of a random variable y is de…ned as 2

= var (y) = E (y

Ey)2

= Ey 2

(Ey)2 :

Notice that the variance is the function of two moments, Ey 2 and Ey: We can calculate the variance of the sample mean b: It is convenient to de…ne the centered observations ui = yi which have mean zero and variance 2 . Then n

^

1X = ui n i=1

and var (^ ) = E (^

)2 = E

n 1X ui n i=1

1 n n n n X 1 XX 1 X @1 uj A = 2 E (ui uj ) = 2 n n n

!0

i=1 j=1

j=1

where the second-to-last inequality is because E (ui uj ) = to independence.

2

i=1

2

=

1 n

2

for i = j yet E (ui uj ) = 0 for i 6= j due

10

CHAPTER 2. MOMENT ESTIMATION Theorem 2.4.1 If

2

< 1 then var (b) =

1 2 n

This result links the variance of the estimator b with the variance of the individual observation yi and with the sample size n: In particular, var (b) is proportional to 2 ; and inversely proportional to n and thus decreases as n increases.

2.5

Convergence in Probability

In Theorem 2.4.1 we showed that the variance of b decreases with the sample size n: This implies that the sampling distribution of b concentrates as the sample size increases. We now give a formal de…nition. De…nition 2.5.1 A random variable zn 2 R converges in probability p to z as n ! 1; denoted zn ! z; if for all > 0; lim Pr (jzn

n!1

zj

) = 1:

(2.1)

The de…nition looks quite abstract, but it formalizes the concept of a distribution concentrating about a point. The event fjzn zj g is the event that zn is within of the point z: Pr (jzn zj ) is the probability of this event – that zn is within of the point z. The statement (2.1) is that this probability approaches 1 as the sample size n increases. The de…nition of convergence in probability requires that this holds for any : So even for very small intervals about z; the distribution of zn concentrates within this interval for large n: p When zn ! z we call z the probability limit (or plim) of zn . Two comments about the notation are worth mentioning. First, it is conventional to write the p convergence symbol as ! where the “p” above the arrow indicates that the convergence is “in probability”. You should try and adhere to this notation, and not simply write zn ! z. Second, it is also important to include the phrase “as n ! 1” to be speci…c about how the limit is obtained. Students often confuse convergence in probability with convergence in expectation: Ezn ! Ez

(2.2)

but these are distinct concepts. Neither (2.1) nor (2.2) implies the other. To see the distinction it might be helpful to think through a stylized example. Consider a discrete random variable zn which takes the value 0 with probability n 1 and the value an 6= 0 with probability 1 n 1 , or Pr (zn = an ) = Pr (zn = 0) =

1 n n

(2.3) 1 n

:

In this example the probability distribution of zn concentrates at zero as n increases. You can p check that zn ! 0 as n ! 1; regardless of the sequence an : In this example we can also calculate that the expectation of zn is Ezn =

an : n

11

CHAPTER 2. MOMENT ESTIMATION

Despite the fact that zn converges in probability to zero, its expectation will not decrease to zero unless an =n ! 0: If an diverges to in…nity at a rate equal to n (or faster) then Ezn will not converge p to zero. For example, if an = n; then Ezn = 1 for all n; even though zn ! 0: This example might seem a bit arti…cial, but the point is that the concepts of convergence in probability and convergence in expectation are distinct, so it is important not to confuse one with the other. Another common source of confusion with the notation surrounding probability limits is that p the expression to the right of the arrow \ !” must be free of dependence on the sample size n: p Thus expressions of the form “zn ! cn ” are notationally meaningless and must not be used.

2.6

Weak Law of Large Numbers

As we mentioned in the two previous sections, the variance of the sample mean decreases to zero as the sample size increases. We now show that this implies that the sample mean converges in probability to the population mean. When y has a …nite variance there is a fairly straightforward proof by applying Chebyshev’s inequality (B.23). The latter states that for any random variable zn and constant > 0 Ezn j > )

Pr (jzn Set zn = b; for which Ezn =

and var(zn ) =

1 2 n

var(zn ) 2

:

(by Theorems 2.3.1 and 2.4.1). Then 2

j> )

Pr (jb

n

2:

For …xed 2 and ; the bound on the right-hand-side shrinks to zero as n ! 1: Thus the probability that b is within of approaches 1 as n gets large, so b converges in probability to . We have shown that the sample mean b converges in probability to the population mean . This result is called the weak law of large numbers. Our derivation assumed that y has a …nite variance, but this is not necessary. It is only necessary for y to have a …nite mean. Theorem 2.6.1 Weak Law of Large Numbers (WLLN) Under Assumption 2.2.1, if E jyj < 1; then as n ! 1, n

y=

1X p yi ! E(yi ): n i=1

The proof of Theorem 2.6.1 is presented in Section 2.15. The WLLN shows that the estimator b = y converges in probability to the true population mean . An estimator which converges in probability to the population value is called consistent. De…nition 2.6.1 An estimator ^ of a parameter

p is consistent if ^ !

as n ! 1:

Consistency is a good property for an estimator to possess. It means that for any given data distribution; there is a sample size n su¢ciently large such that the estimator ^ will be arbitrarily close to the true value with high probability. Unfortunately it does not mean that ^ will actually be close to in a given …nite sample, but it is a minimal property for an estimator to be considered a “good” estimator.

12

CHAPTER 2. MOMENT ESTIMATION Theorem 2.6.2 Under Assumption 2.2.1, if E jyj < 1; then b = y is consistent for the population mean :

Almost Sure Convergence and the Strong Law* Convergence in probability is sometimes called weak convergence. A related concept is almost sure convergence, also known as strong convergence. (In probability theory the term “almost sure” means “with probability equal to one”. An event which is random but occurs with probability equal to one is said to be almost sure.)

De…nition 2.6.2 A random variable zn 2 R converges almost surely a:s: to z as n ! 1; denoted zn ! z; if for every > 0 Pr

lim jzn

n!1

zj

= 1:

(2.4)

The convergence (2.4) is stronger than (2.1) because it computes the probability of a limit rather than the limit of a probability. Almost sure convergence is stronger than p a:s: convergence in probability in the sense that zn ! z implies zn ! z. In the example (2.3) of Section 2.5, the sequence zn converges in probability to zero for any sequence an ; but this is not su¢cient for zn to converge almost surely. In order for zn to converge to zero almost surely, it is necessary that an ! 0. In the random sampling context the sample mean can be shown to converge almost surely to the population mean. This is called the strong law of large numbers.

Theorem 2.6.3 Strong Law of Large Numbers (SLLN) Under Assumption 2.2.1, if E jyj < 1; then as n ! 1, n

y=

1 X a:s: yi ! E(yi ): n i=1

The proof of the SLLN is technically quite advanced so is not presented here. For a proof see Billingsley (1995, Section 22) or Ash (1972, Theorem 7.2.5). The WLLN is su¢cient for most purposes in econometrics, so we will not use the SLLN in this text.

2.7

Vector-Valued Moments

Our preceding discussion focused on the case where y is real-valued (a scalar), but nothing important changes if we generalize to the case where y 2 Rm is a vector. To …x notation, the

13

CHAPTER 2. MOMENT ESTIMATION elements of y are

0

B B y=B @

y1 y2 .. .

1

C C C: A

ym The population mean of y is just the vector of marginal means 1 0 E (y1 ) B E (y2 ) C C B = E(y) = B C: .. A @ . E (ym )

When working with random vectors y it is convenient to measure their magnitude with the Euclidean norm 2 1=2 kyk = y12 + + ym :

This is the classic Euclidean length of the vector y. Notice that kyk2 = y 0 y:

It turns out that it is equivalent to describe …niteness of moments in terms of the Euclidean norm of a vector or all individual components. Theorem 2.7.1 For y 2 Rm ; E kyk < 1 if and only if E jyj j < 1 for j = 1; :::; m:

Theorem 2.7.1 implies that the components of The m m variance matrix of y is V = var (y) = E (y

are …nite if and only if E kyk < 1. ) (y

)0 :

V is often called a variance-covariance matrix. You can show that the elements of V are …nite if E kyk2 < 1: A random sample fy 1 ; :::; y n g consists of n observations of independent and identically draws from the distribution of y: (Each draw is an m-vector.) The vector sample mean 1 0 y1 n B y C 1X B 2 C y= yi = B . C n @ .. A i=1

ym

is the vector of means of the individual variables. Convergence in probability of a vector can be de…ned as convergence in probability of all elep p ments in the vector. Thus y ! if and only if y j ! j for j = 1; :::; m: Since the latter holds if E jyj j < 1 for j = 1; :::; m; or equivalently E kyk < 1; we can state this formally as follows. Theorem 2.7.2 Weak Law of Large Numbers (WLLN) for random vectors Under Assumption 2.2.1, if E jyj < 1; then as n ! 1, n

y=

1X p y i ! E(y i ): n i=1

14

CHAPTER 2. MOMENT ESTIMATION

2.8

Convergence in Distribution

The WLLN is a useful …rst step, but does not give an approximation to the distribution of an estimator. A large-sample or asymptotic approximation can be obtained using the concept of convergence in distribution.

De…nition 2.8.1 Let z n be a random vector with distribution Fn (u) = Pr (z n

u) : We

d

say that z n converges in distribution to z as n ! 1, denoted z n ! z; if for all u at which F (u) = Pr (z u) is continuous, Fn (u) ! F (u) as n ! 1:

d

When z n ! z, it is common to refer to z as the asymptotic distribution or limit distribution of z n . When the limit distribution z is degenerate (that is, Pr (z = c) = 1 for some c) we can write p

d

the convergence as z n ! c, which is equivalent to convergence in probability, z n ! c. The typical path to establishing convergence in distribution is through the central limit theorem (CLT), which states that a standardized sample average converges in distribution to a normal random vector.

Theorem 2.8.1 Lindeberg–Lévy Central Limit Theorem (CLT). Under Assumption 2.2.1, if E kyk2 < 1; then as n ! 1 p

n

n (y n

1 X (y i )= p n i=1

where

= Ey and V = E (y

) (y

d

) ! N (0; V ) )0 :

p The standardized sum z n = n (y n ) has mean zero and variance V . What the CLT adds is that the variable z n is also approximately normally distributed, and that the normal approximation improves as n increases. The CLT is one of the most powerful and mysterious results in statistical theory. It shows that the simple process of averaging induces normality. The …rst version of the CLT (for the number of heads resulting from many tosses of a fair coin) was established by the French mathematician Abraham de Moivre in 1733. This was extended to cover an approximation to the binomial distribution in 1812 by Pierre-Simon Laplace, and the most general statements are credited to the Russian mathematician Aleksandr Lyapunov (1901) and the Finnish mathematician Jarl Waldemar Lindeberg (1922). The above statement is known as the classic (or Lindeberg-Lévy) CLT due to contributions by Lindeberg (1920) and the French mathematician Paul Pierre Lévy. A more general version which does not require the restriction to identical distributions was provided by Lindeberg (1922).

15

CHAPTER 2. MOMENT ESTIMATION Theorem 2.8.2 Lindeberg Central Limit Theorem (CLT). Suppose that yi are independent but not necessarily identically distributed …nite Pn with 2 2 2 2 means i = Eyi and variances i = E (yi i ) : Set n = i=1 i : If for all " > 0 n 1 X 2 E (yi " n) = 0 (2.5) lim 2 i ) 1 (jyi ij n!1

n i=1

then

n 1 X n i=1

(yi

i)

d

! N (0; 1)

Equation (2.5) is known as Lindeberg’s condition. A standard method to verify (2.5) is via Lyapunov’s condition: For some > 0 lim

n!1

1 2+ n

n X

2+ i)

E (yi

=0

(2.6)

i=1

It is easy to verify that (2.6) implies (2.5), and (2.6) is often easy to verify. For example, if 3 supi E (yi < 1 and inf i 2i c > 0 then i) n 1 X 3 n i=1

n

3 i)

E (yi

(nc)3=2

!0

so (2.6) is satis…ed.

2.9

Functions of Moments

We now expand our investigation and consider estimation of parameters which can be written as a continuous function of . That is, the parameter of interest is the vector of functions = g( )

(2.7)

where g : Rm ! Rk : As one example, the geometric mean of wages w is = exp (E (log (w)))

(2.8)

which is (2.7) with g(u) = exp (u) and

= E (log (w)) : As another example, the skewness of the wage distribution is Ew)3

E (w

sk =

E (w

Ew)2

3=2

= g Ew; Ew2 ; Ew3 where w = wage and g(

1;

2;

3)

=

3

3 2

In this case we can set

3 2 1+2 1 : 2 3=2 1

1 w y = @ w2 A w3 0

(2.9)

16

CHAPTER 2. MOMENT ESTIMATION so that

1 Ew = @ Ew2 A : Ew3 0

(2.10)

The parameter = g ( ) is not a population moment, so it does not have a direct moment estimator. Instead, it is common to use a plug-in estimate formed by replacing the unknown with its point estimate b so that b = g (b ) : (2.11) Again, the hat “^” indicates that b is a sample estimate of : For example, the plug-in estimate of the geometric mean of the wage distribution from (2.8) is b = exp(b) with

n

b=

1X log (wagei ) : n i=1

The plug-in estimate of the skewness of the wage distribution is 1 Pn w)3 i=1 (wi n c sk = 3=2 1 Pn w)2 i=1 (wi n =

where

b3

3b2 b1 + 2b31 b21

b2

bj =

3=2

n

1X j wi : n i=1

A useful property is that continuous functions are limit-preserving.

p

Theorem 2.9.1 Continuous Mapping Theorem (CMT). If z n ! c p as n ! 1 and g ( ) is continuous at c; then g(z n ) ! g(c) as n ! 1.

The proof of Theorem 2.9.1 is given in Section 2.15. p For example, if zn ! c as n ! 1 then p

zn + a ! c + a p

azn ! ac p

zn2 ! c2

as the functions g (u) = u + a; g (u) = au; and g (u) = u2 are continuous. Also a zn

p

!

a c

if c 6= 0: The condition c 6= 0 is important as the function g(u) = a=u is not continuous at u = 0: We need the following assumption in order for b to be consistent for .

17

CHAPTER 2. MOMENT ESTIMATION Theorem 2.9.2 Under Assumption 2.2.1, if E jyj < 1 and g (u) is continuous at u = , then b = g (b ) p! g ( ) =

as n ! 1; and thus b is consistent for

:

To apply Theorem 2.9.2 it is necessary to check if the function g is continuous at . In our …rst example g(u) = exp (u) is continuous everywhere. It therefore follows from Theorem 2.7.2 and Theorem 2.9.2 that if E jlog (wage)j < 1 then as n ! 1 p

b ! :

2 > 0; In our second example g de…ned in (2.9) is continuous for all such that var(w) = 2 1 3 which holds unless w has a degenerate distribution. Thus if E jwj < 1 and var(w) > 0 then as n!1 c p! sk: sk

2.10

Delta Method

In this section we introduce two tools – an extended version of the CMT and the Delta Method – which allow us to calculate the asymptotic distribution of the parameter estimate b . We …rst present an extended version of the continuous mapping theorem which allows convergence in distribution.

Theorem 2.10.1 Continuous Mapping Theorem d If z n ! z as n ! 1 and g : Rm ! Rk has the set of discontinuity points d Dg such that Pr (z 2 Dg ) = 0; then g(z n ) ! g(z) as n ! 1.

For a proof of Theorem 2.10.1 see Theorem 2.3 of van der Vaart (1998). It was …rst proved by Mann and Wald (1943) and is therefore sometimes referred to as the Mann-Wald Theorem Theorem 2.10.1 allows the function g to be discontinuous only if the probability at being at a discontinuity point is zero. For example, the function g(u) = u 1 is discontinuous at u = 0; but if d

d

zn ! z N (0; 1) then Pr (z = 0) = 0 so zn 1 ! z 1 : A special case of the Continuous Mapping Theorem is known as Slutsky’s Theorem.

Theorem 2.10.2 Slutsky’s Theorem p d If zn ! z and cn ! c as n ! 1, then d

1. zn + cn ! z + c d

2. zn cn ! zc 3.

zn cn

d

!

z if c 6= 0 c

18

CHAPTER 2. MOMENT ESTIMATION

Even though Slutsky’s Theorem is a special case of the CMT, it is a useful statement as it focuses on the most common applications – addition, multiplication, and division. Despite the fact that the plug-in estimator b is a function of b for which we have an asymptotic distribution, Theorem 2.10.1 does not directly give us an asymptotic distribution for b : This is p because b = g (b ) is written as a function of b , not of the standardized sequence n (b ): We need an intermediate step – a …rst order Taylor series expansion. This step is so critical to statistical theory that it has its own name – The Delta Method.

Theorem 2.10.3 Delta Method: p d If n ( n is m 1; and g( ) : Rm ! Rk ; k 0 ) ! ; where continuously di¤erentiable in a neighborhood of then as n ! 1 p

where G( ) =

@ @

n (g (

n(

0 ):

0)

n

0 ))

g(

g( )0 and G = G( p

where V is m

n)

d

! G0

m; is

(2.12)

In particular, if

d

! N (0; V )

m; then as n ! 1 p

n (g (

n)

g(

0 ))

d

! N 0; G0 V G :

(2.13)

The Delta Method allows us to complete our derivation of the asymptotic distribution of the estimator b of . Relative to consistency, it requires the stronger smoothness condition that g( ) is continuously di¤erentiable. Now by combining Theorems 2.8.1 and 2.10.3 we can …nd the asymptotic distribution of the plug-in estimator b . Theorem 2.10.4 Under Assumption 2.2.1, if E kyk2 < 1; and G (u) = @ g (u)0 is continuous in a neighborhood of u = , then as n ! 1 @u p where G = G ( ) :

2.11

d

n b

! N 0; G0 V G

Stochastic Order Symbols

It is convenient to have simple symbols for random variables and vectors which converge in probability to zero or are stochastically bounded. In this section we introduce some of the most commonly found notation. It might be useful to review the common notation for non-random convergence and boundedness. Let xn and an ; n = 1; 2; :::; be a non-random sequences. The notation xn = o(1)

19

CHAPTER 2. MOMENT ESTIMATION (pronounced “small oh-one”) is equivalent to an ! 0 as n ! 1. The notation xn = o(an ) is equivalent to an 1 xn ! 0 as n ! 1: The notation xn = O(1)

(pronounced “big oh-one”) means that xn is bounded uniformly in n : there exists an M < 1 such that jxn j M for all n: The notation xn = O(an ) is equivalent to an 1 xn = O(1): We now introduce similar concepts for sequences of random variables. Let zn and an ; n = 1; 2; ::: be sequences of random variables. (In most applications, an is non-random.) The notation zn = op (1) p

(“small oh-P-one”) means that zn ! 0 as n ! 1: We also write zn = op (an ) if an 1 zn = op (1): For example, for any consistent estimator b for b=

we can write

+ op (1)

Similarly, the notation zn = Op (1) (“big oh-P-one”) means that zn is bounded in probability. Precisely, for any " > 0 there is a constant M" < 1 such that lim Pr (jzn j > M" )

":

n!1

Furthermore, we write zn = Op (an ) if an 1 zn = Op (1): Op (1) is weaker than op (1) in the sense that zn = op (1) implies zn = Op (1) but not the reverse. However, if zn = Op (an ) then zn = op (bn ) for any bn such that an =bn ! 0: d

If a random vector converges in distribution z n ! z (for example, if z N (0; V )) then z n = Op (1): It follows that for estimators b which satisfy the convergence of Theorem 2.10.4 then we can write b = + Op (n 1=2 ):

Another useful observation is that a random sequence with a bounded moment is stochastically bounded.

Theorem 2.11.1 If z n is a random vector which satis…es E kz n k = O (an ) for some sequence an and

> 0; then z n = Op (a1=r n ): 1=r

Similarly, E kz n k = o (an ) implies z n = op (an ):

20

CHAPTER 2. MOMENT ESTIMATION

This can be shown using Markov’s inequality (B.21). The assumptions imply that there is some M an 1= : Then M < 1 such that E kz n k M an for all n: For any " set B = " Pr (kz n k > B) = Pr kz n k >

M an "

" E kz n k M an

"

as required. There are many simple rules for manipulating op (1) and Op (1) sequences which can be deduced from the continuous mapping theorem or Slutsky’s Theorem. For example, op (1) + op (1) = op (1) op (1) + Op (1) = Op (1) Op (1) + Op (1) = Op (1) op (1)op (1) = op (1) op (1)Op (1) = op (1) Op (1)Op (1) = Op (1)

2.12

Uniform Stochastic Bounds*

For some applications it can be useful to obtain the stochastic order of the random variable max jyi j :

1 i n

This is the magnitude of the largest observation in the sample fy1 ; :::; yn g: If the support of the distribution of yi is unbounded, then as the sample size n increases, the largest observation will also tend to increase. It turns out that there is a simple characterization.

Theorem 2.12.1 Under Assumption 2.2.1, if E jyjr < 1; then as n ! 1 n

1=r

p

max jyi j ! 0

1 i n

Equivalently, max jyi j = op (n1=r ):

1 i n

(2.14)

Theorem 2.12.1 says that the largest observation will diverge at a rate slower than n1=r . As r increases this rate decreases. Thus the higher the moment, the slower the rate of divergence of the largest observation. To simplify the notation, we write (2.14) as yi = op (n1=r ) uniformly in 1 i n. It is important to understand when the Op or op symbols are applied to subscript i random variables we typically mean uniform convergence in the sense of (2.14). Theorem 2.12.1 applies to random vectors. If E kykr < 1 then max ky i k = op (n1=r ):

1 i n

21

CHAPTER 2. MOMENT ESTIMATION

We now prove Theorem 2.12.1. Take any : The event max1 i n jyi j > n1=r means that at S least the jyi j exceeds n1=r ; which is the same as the event ni=1 jyi j > n1=r or equivalently Sn one of r r ng : Since the probability of the union of events is smaller than the sum of the i=1 fjyi j > probabilities, ! n [ r 1=r r Pr n max jyi j > = Pr fjyi j > ng 1 i n

i=1

n X i=1

Pr (jyi jr > n r )

n 1 X E (jyi jr 1 (jyi jr > n r )) n r i=1

=

1

r r E (jyi j

1 (jyi jr > n r ))

where the second inequality is the strong form of Markov’s inequality (Theorem B.22) and the …nal equality is since the yi are iid. Since E jyjr < 1 this …nal expectation converges to zero as n ! 1: This is because Z r E jyi j = jyjr dF (y) < 1

implies

E (jyi jr 1 (jyi jr > c)) = as c ! 1: We have established that n

2.13

1=r

max1

Z

r

jyj >c

i n jyi j

jyjr dF (y) ! 0 p

! 0; as required.

Semiparametric E¢ciency

In this section we argue that the sample mean b and plug-in estimator b = g (b ) are e¢cient estimators of the parameters and . Our demonstration is based on the rich but technically challenging theory of semiparametric e¢ciency bounds. An excellent accessible review has been provided by Newey (1990). We will also appeal to the asymptotic theory of maximum likelihood estimation (see Section B.11). We start by examining the sample mean b ; for the asymptotic e¢ciency of b will follow from that of b : Recall, we know that if E kyk2 < 1 then the sample mean has the asymptotic distribution p d n (b ) ! N (0; V ) : We want to know if b is the best feasible estimator, or if there is another estimator with a smaller asymptotic variance. While it seems intuitively unlikely that another estimator could have a smaller asymptotic variance, how do we know that this is not the case? When we ask if b is the best estimator, we need to be clear about the class of models – the class of permissible distributions. For estimation of the mean of the distribution of y the broadest conceivable class is L1 = fF : E kyk < 1g : This class is too broad n for our current purposes, as n o b is not asymptotically N (0; V ) for all F 2 L1 : A more realistic choice is L2 = F : E kyk2 < 1 – the class of …nite-variance distributions. When we seek an e¢cient estimator of the mean in the class of models L2 what we are seeking is the best estimator, given that all we know is that F 2 L2 : To show that the answer is not immediately obvious, it might be helpful to review a setting where the sample mean is ine¢cient. Suppose that y 2 R has the double exponential denp sity f (y j ) = 2 1=2 exp jy j 2 : Since var (y) = 1 we see that the sample mean satp d is…es n (^ ) ! N (0; 1). In this model the maximum likelihood estimator (MLE) ~ for is the sample median. Recall from the theory of maximum likelhood that the MLE satis…es

22

CHAPTER 2. MOMENT ESTIMATION

p log f (y j ) = 2 sgn (y ) is the score. We can p d calculate that ES 2 = 2 and thus conclude that n (~ ) ! N (0; 1=2) : The asymptotic variance of the MLE is one-half that of the sample mean. Thus when the true density is known to be double exponential the sample mean is ine¢cient. But the estimator which achieves this improved e¢ciency – the sample median – is not generically consistent for the population mean. It is inconsistent if the density is asymmetric or skewed. So the improvement comes at a great cost. Another way of looking at this p is that the sample median is e¢cient in the class of densities f (y j ) = 2 1=2 exp jy j 2 but unless it is known that this is the correct distribution class this knowledge is not very useful. The relevant question is whether or not the sample mean is e¢cient when the form of the distribution is unknown. We call this setting semiparametric as the parameter of interest (the mean) is …nite dimensional while the remaining features of the distribution are unspeci…ed. In the semiparametric context an estimator is called semiparametrically e¢cient if it has the smallest asymptotic variance among all semiparametric estimators. The mathematical trick is to reduce the semiparametric model to a set of parametric “submodels”. The Cramer-Rao variance bound can be found for each parametric submodel. The variance bound for the semiparametric model (the union of the submodels) is then de…ned as the supremum of the individual variance bounds. R Formally, suppose that the true density of y is the unknown function f (y) with mean = Ey = yf (y)dy: A parametric submodel for f (y) is a density f (y j ) which is a smooth function of a parameter , and there is a true value 0 such that f (y j 0 ) = f (y): The index indicates the submodels. The equality f (y j 0 ) = f (y) means that the submodel class passes through the true density, so the submodel is a true model. The class of submodels and R parameter 0 depend on the true density f: In the submodel f (y j ) ; the mean is ( ) = yf (y j ) dy which varies with the parameter . Let 2 @ be the class of all submodels for f: Since each submodel is parametric we can calculate the e¢ciency bound for estimation of within this submodel. Speci…cally, given the density f (y j ) its likelihood score is p

d

) ! N 0; ES 2

n (~

1

where S =

S =

@ @

@ log f (y j @

0) ; 1

so the Cramer-Rao lower bound for estimation of is ES S 0 : De…ning M = @@ ( by Theorem B.11.5 the Cramer-Rao lower bound for estimation of within the submodel 0

1

0 0) ;

is

M . V = M ES As V is the e¢ciency bound for the submodel class f (y j ) ; no estimator can have an asymptotic variance smaller than V for any density f (y j ) in the submodel class, including the true density f . This is true for all submodels : Thus the asymptotic variance of any semiparametric estimator cannot be smaller than V for any conceivable submodel. Taking the supremum of the Cramer-Rao bounds lower from all conceivable submodels we de…ne2 S0

V = sup V : 2@

The asymptotic variance of any semiparametric estimator cannot be smaller than V , since it cannot be smaller than any individual V : We call V the semiparametric asymptotic variance bound or semiparametric e¢ciency bound for estimation of , as it is a lower bound on the asymptotic variance for any semiparametric estimator. If the asymptotic variance of a speci…c semiparametric estimator equals the bound V we say that the estimator is semiparametrically e¢cient. For many statistical problems it is quite challenging to calculate the semiparametric variance bound. However, in some cases there is a simple method to …nd the solution. Suppose that 2

It is not obvious that this supremum exists, as V is a matrix so there is not a unique ordering of matrices. However, in many cases (including the ones we study) the supremum exists and is unique.

23

CHAPTER 2. MOMENT ESTIMATION

we can …nd a submodel 0 whose Cramer-Rao lower bound satis…es V 0 = V where V is the asymptotic variance of a known semiparametric estimator. In this case, we can deduce that V = V 0 = V . Otherwise there would exist another submodel 1 whose Cramer-Rao lower bound satis…es V 0 < V 1 but this would imply V < V 1 which contradicts the Cramer-Rao Theorem. We now …nd this submodel for the sample mean b : Our goal is to …nd a parametric submodel whose Cramer-Rao bound for is V : This can be done by creating a tilted version of the true density. Consider the parametric submodel 0

V

where f (y) is the true density and = Ey: Note that Z Z f (y j ) dy = f (y)dy + 0 V

1

f (y j ) = f (y) 1 +

1

Z

(y

)

(2.15)

f (y) (y

) dy = 1

and for all close to zero f (y j ) 0: Thus f (y j ) is a valid density function. It is a parametric submodel since f (y j 0 ) = f (y) when 0 = 0: This parametric submodel has the mean Z ( ) = yf (y j ) dy Z Z = yf (y)dy + f (y)y (y )0 V 1 dy =

+

which is a smooth function of : Since @ @ log f (y j ) = log 1 + @ @ it follows that the score function for S =

0

V

1

(y

1

V 1 (y ) 0 1 1 + V (y )

is

@ log f (y j @

0)

=V

By Theorem B.11.3 the Cramer-Rao lower bound for E S S0

) =

= V

1

E (y

1

(y

):

(2.16)

is

) (y

)0 V

1

1

=V:

(2.17)

The Cramer-Rao lower bound for ( ) = + is also V , and this equals the asymptotic variance of the moment estimator b : This was what we set out to show. In summary, we have shown that in the submodel (2.15) the Cramer-Rao lower bound for estimation of is V which equals the asymptotic variance of the sample mean. This establishes the following result.

Proposition 2.13.1 In the class of distributions F 2 L2 ; the semiparametric variance bound for estimation of is V = var(yi ); and the sample mean b is a semiparametrically e¢cient estimator of the population mean . We call this result a proposition rather than a theorem as we have not attended to the regularity conditions. It is a simple matter to extend this result to the plug-in estimator b = g (b ). We know from Theorem 2.10.4 that if E kyk2 < 1 and g (u) is continuously di¤erentiable at u = then the plug-in

24

CHAPTER 2. MOMENT ESTIMATION

p d ! N (0; G0 V G) : We therefore consider the estimator has the asymptotic distribution n b n o class of distributions L2 (g) = F : E kyk2 < 1; g (u) is continuously di¤erentiable at u = Ey :

For example, if = 1 = 2 where 1 = Ey1 and 2 = Ey2 then L2 (g) = F : Ey12 < 1; Ey22 < 1; and Ey2 6= 0 : For any submodel the Cramer-Rao lower bound for estimation of = g ( ) is G0 V G by Theorem B.11.5. For the submodel (2.15) this bound is G0 V G which equals the asymptotic variance of b from Theorem 2.10.4. Thus b is semiparametrically e¢cient. Proposition 2.13.2 In the class of distributions F 2 L2 (g) the semiparametric variance bound for estimation of = g ( ) is G0 V G; and the plug-in estimator b = g (b ) is a semiparametrically e¢cient estimator of .

The result in Proposition 2.13.2 is quite general. Smooth functions of sample moments are e¢cient estimators for their population counterparts. This is a very powerful result, as most econometric estimators can be written (or approximated) as smooth functions of sample means.

2.14

Expectation*

For any random variable y we de…ne the mean or expectation Ey as follows. If y is discrete, Ey =

1 X

j

Pr (y =

j) ;

j=1

and if y is continuous with density f Ey =

Z

1

yf (y)dy:

1

We can unify these de…nitions by writing the expectation as the Lebesgue integral with respect to the distribution function F Z 1 ydF (y): Ey = 1

The mean is well de…ned and …nite if E jyj =

Z

1 1

jyj dF (y) < 1:

If this does not hold, we evaluate I1 =

Z

0

I2 =

1

Z

ydF (y) 0

ydF (y) 1

If I1 = 1 and I2 < 1 then we de…ne Ey = 1: If I1 < 1 and I2 = 1 then we de…ne Ey = 1: If both I1 = 1 and I2 = 1 then Ey is unde…ned. If = Ey is well de…ned we say that is identi…ed, meaning that the parameter is uniquely determined by the distribution of the observed variables. The demonstration that the parameters of an econometric model are identi…ed is an important precondition for estimation. Typically, identi…cation holds under a set of restrictions, and an identi…cation theorem carefully describes a

25

CHAPTER 2. MOMENT ESTIMATION

set of such conditions which are su¢cient for identi…cation. In the case of the mean ; a su¢cient condition for identi…cation is E jyj < 1: The mean of y is …nite if E jyj < 1: More generally, y has a …nite r’th moment if E jyjr < 1: It is common in econometric theory to assume that the variables, or certain transformations of the variables, have …nite moments of a certain order. How should we interpret this assumption? How restrictive is it? One way to visualize the importance is to consider the class of Pareto densities given by a 1

f (y) = ay

;

y > 1:

The parameter a of the Pareto distribution indexes the rate of decay of the tail of the density. Larger a means that the tail declines to zero more quickly. See the …gure below where we show the Pareto density for a = 1 and a = 2: The parameter a also determines which moments are …nite. We can calculate that 8 R a 1 r a 1 > if r < a dy = < a 1 y a r r E jyj = > : 1 if r a Thus to allow for stricter …nite moments (larger r) we need to restrict the class of permissible densities (require larger a).

f(y)

2.0

a=2 1.5

1.0

a=1

0.5

0.0 1

2

3

4

y

Pareto Densities, a = 1 and a = 2 Thus, broadly speaking, the restriction that y has a …nite r’th moment means that the tail of y’s density declines to zero faster than y r 1 : The faster decline of the tail means that the probability of observing an extreme value of y is a more rare event. It is helpful to know that the existence of …nite moments is monotonic. Liapunov’s Inequality (B.20) implies that if E jyjp < 1 for some p > 0, then E jyjr < 1 for all 0 r p: For example, Ey 2 < 1 implies E jyj < 1 and thus both the mean and variance of y are …nite.

2.15

Technical Proofs*

In this section we provide proofs of some of the more technical points in the chapter. These proofs may only be of interest to more mathematically inclined. Proof of Theorem 2.6.1: Without loss of generality, we can assume E(yi ) = 0 by recentering yi on its expectation.

CHAPTER 2. MOMENT ESTIMATION

26

We need to show that for all > 0 and > 0 there is some N < 1 so that for all n : Fix and : Set " = =3: Pick C < 1 large enough so that Pr (jyj > )

N;

E (jyi j 1 (jyi j > C))

"

(2.18)

(where 1 ( ) is the indicator function) which is possible since E jyi j < 1: De…ne the random variables wi = yi 1 (jyi j

E (yi 1 (jyi j

C)

C))

E (yi 1 (jyi j > C))

zi = yi 1 (jyi j > C)

so that y =w+z and E jyj

E jwj + E jzj :

(2.19)

We now show that sum of the expectations on the right-hand-side can be bounded below 3": First, by the Triangle Inequality (A.12) and the Expectation Inequality (B.15), E jzi j = E jyi 1 (jyi j > C)

E (yi 1 (jyi j > C))j

E jyi 1 (jyi j > C)j + jE (yi 1 (jyi j > C))j 2E jyi 1 (jyi j > C)j 2";

(2.20)

and thus by the Triangle Inequality (A.12) and (2.20) n

E jzj = E

n

1X zi n

1X E jzi j n

i=1

2":

(2.21)

i=1

Second, by a similar argument jwi j = jyi 1 (jyi j jyi 1 (jyi j

2 jyi 1 (jyi j

E (yi 1 (jyi j

C)

C)j + jE (yi 1 (jyi j

C))j C))j

C)j

2C

(2.22)

where the …nal inequality is (2.18). Then by Jensen’s Inequality (B.12), the fact that the wi are iid and mean zero, and (2.22), (E jwj)2 the …nal inequality holding for n together show that

E jwj2 =

Ewi2 4C 2 = n n

4C 2 ="2 = 36C 2 = E jyj

2 2 .

"2

(2.23)

Equations (2.19), (2.21) and (2.23)

3"2

(2.24)

as desired. Finally, by Markov’s Inequality (B.21) and (2.24), Pr (jyj > )

E jyj

3"

= ;

the …nal equality by the de…nition of ": We have shown that for any ; as needed. n 36C 2 = 2 2 ; Pr (jyj > )

> 0 and

> 0 then for all

27

CHAPTER 2. MOMENT ESTIMATION Proof of Theorem 2.7.1: By Loève’s cr Inequality (A.19) 0 11=2 m X kyk = @ yj2 A

m X

j=1

j=1

jyj j :

Thus if E jyj j < 1 for j = 1; :::; m; then

E kyk

m X j=1

E jyj j < 1:

For the reverse inequality, the Euclidean norm of a vector is larger than the length of any individual component, so for any j; jyj j kyk : Thus, if E kyk < 1; then E jyj j < 1 for j = 1; :::; m: Proof of Theorem 2.8.1: The moment bound Ey 0i y i < 1 is su¢cient to guarantee that the elements of and V are well de…ned and …nite. Without loss of generality, it is su¢cient to consider the case = 0: p Our proof method is to calculate the characteristic function of ny n and show that it converges pointwise to the characteristic function of N (0; V ) : By Lévy’s Continuity Theorem (see Van der p Vaart (2008) Theorem 2.13) this is su¢cient to established that ny n converges in distribution to N (0; V ) : For 2 Rm ; let C ( ) = E exp i 0 y i denote the characteristic function of y i and set c ( ) = log C( ). Since y i has two …nite moments the …rst and second derivatives of C( ) are continuous in : They are @ C( ) = iE y i exp i 0 y i @ @2 @ @ When evaluated at

0 C(

) = i2 E y i y 0i exp i 0 y i

:

=0 C(0) = 1 @ C(0) = iE (y i ) = 0 @ @2 @ @

0 C(0)

E y i y 0i =

=

V:

Furthermore, c ( ) = c

( ) =

so when evaluated at

@ @ c( ) = C( ) 1 C( ) @ @ 2 @2 1 @ c( ) = C( ) C( ) @ @ 0 @ @ 0

2

C( )

@ @ C( ) C( ) @ @ 0

=0 c(0) = 0 c (0) = 0 c (0) =

V:

By a second-order Taylor series expansion of c( ) about c( ) = c(0) + c (0)0 +

1 2

0

c

(

) =

= 0; 1 2

0

c

(

)

(2.25)

28

CHAPTER 2. MOMENT ESTIMATION

where lies on the line segment joining 0 and : p p We now compute Cn ( ) = E exp i 0 ny n ; the characteristic function of ny n : By the properties of the exponential function, the independence of the y i ; the de…nition of c( ) and (2.25) ! n 1 X 0 log Cn ( ) = log E exp i p yi n i=1

n Y

1 exp i p n

0

yi

1 E exp i p n

0

yi

1 log E exp i p n

0

= log E

i=1

= log

n Y i=1

=

n X i=1

p

= nc =

1 2

0

c

yi

n (

n)

p where n lies on the line segment joining 0 and = n: Since c ( n ) ! c (0) = V: We thus …nd that as n ! 1; log Cn ( ) !

1 2

0

1 2

0

and Cn ( ) ! exp

n

! 0 and c

( ) is continuous,

V

V

which is the characteristic function of the N (0; V ) distribution. This completes the proof. Proof of Theorem 2.9.1: Since g is continuous at c; for all " > 0 we can …nd a > 0 such that if kz n ck < then kg (z n ) g (c)k ": Recall that A B implies Pr(A) Pr(B): Thus p Pr (kg (z n ) g (c)k ") Pr (kz n ck < ) ! 1 as n ! 1 by the assumption that z n ! c: p Hence g(z n ) ! g(c) as n ! 1. Proof of Theorem 2.10.3: By a vector Taylor series expansion, for each element of g; gj ( where

nj

n)

= gj ( ) + gj (

lies on the line segment between

It follows that ajn = gj ( p

jn )

n (g (

gj n)

n

and

jn ) ( n

)

and therefore converges in probability to :

p

! 0: Stacking across elements of g; we …nd

g( )) = (G + an )0

p

n(

d

) ! G0 :

n

p d The convergence is by Theorem 2.10.1, as G + an ! G; n ( continuous. This establishes (2.12) When N (0; V ) ; the right-hand-side of (2.26) equals

n

G0 = G0 N (0; V ) = N 0; G0 V G establishing (2.13).

d

(2.26)

) ! ; and their product is

Chapter 3

Conditional Expectation and Projection 3.1

Introduction

The most commonly applied econometric tool is least-squares estimation, also known as regression. As we will see, least-squares is a tool to estimate an approximate conditional mean of one variable (the dependent variable) given another set of variables (the regressors, conditioning variables, or covariates). In this chapter we abstract from estimation, and focus on the probabilistic foundation of the conditional expectation model and its projection approximation.

3.2

The Distribution of Wages

Suppose that we are interested in wage rates in the United States. Since wage rates vary across workers, we cannot describe wage rates by a single number. Instead, we can describe wages using a probability distribution. Formally, we view the wage of an individual worker as a random variable wage with the probability distribution F (u) = Pr(wage

u):

When we say that a person’s wage is random we mean that we do not know their wage before it is measured, and we treat observed wage rates as realizations from the distribution F: Treating unobserved wages as random variables and observed wages as realizations is a powerful mathematical abstraction which allows us to use the tools of mathematical probability. A useful thought experiment is to imagine dialing a telephone number selected at random, and then asking the person who responds to tell us their wage rate. (Assume for simplicity that all workers have equal access to telephones, and that the person who answers your call will respond honestly.) In this thought experiment, the wage of the person you have called is a single draw from the distribution F of wages in the population. By making many such phone calls we can learn the distribution F of the entire population. When a distribution function F is di¤erentiable we de…ne the probability density function f (u) =

d F (u): du

The density contains the same information as the distribution function, but the density is typically easier to visually interpret.

29

30

Wage Density

0.6 0.5 0.4 0.0

0.1

0.2

0.3

Wage Distribution

0.7

0.8

0.9

1.0

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION

0

10

20

30

40

50

60

70

0

10

Dollars per Hour

20

30

40

50

60

70

80

90

100

Dollars per Hour

Figure 3.1: Wage Distribution and Density. All full-time U.S. workers In Figure 3.1 we display estimates1 of the probability distribution function (on the left) and density function (on the right) of U.S. wage rates in 2009. We see that the density is peaked around $15, and most of the probability mass appears to lie between $10 and $40. These are ranges for typical wage rates in the U.S. population. Important measures of central tendency are the median and the mean. The median m of a continuous2 distribution F is the unique solution to 1 F (m) = : 2 The median U.S. wage ($19.23) is indicated in the left panel of Figure 3.1 by the arrow. The median is a robust3 measure of central tendency, but it is tricky to use for many calculations as it is not a linear operator. For this reason the median is not the dominant measure of central tendency in econometrics. As discussed in Sections 2.2 and 2.14, the expectation or mean of a random variable y with density f is = E (y) Z 1 = uf (u)du: 1

The mean U.S. wage ($23.90) is indicated in the right panel of Figure 3.1 by the arrow. Here we have used the common and convenient convention of using the single character y to denote a random variable, rather than the more cumbersome label wage. The mean is a convenient measure of central tendency because it is a linear operator and arises naturally in many economic models. A disadvantage of the mean is that it is not robust4 especially in the presence of substantial skewness or thick tails, which are both features of the wage 1

The distribution and density are estimated nonparametrically from the sample of 50,742 full-time non-military wage-earners reported in the March 2009 Current Population Survey. The wage rate is constructed as individual wage and salary earnings divided by hours worked. 1 2 g If F is not continuous the de…nition is m = inffu : F (u) 2 3 The median is not sensitive to pertubations in the tails of the distribution. 4 The mean is sensitive to pertubations in the tails of the distribution.

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION

31

Log Wage Density

distribution as can be seen easily in the right panel of Figure 3.1. Another way of viewing this is that 64% of workers earn less that the mean wage of $23.90, suggesting that it is incorrect to describe the mean as a “typical” wage rate.

1

2

3

4

5

6

Log Dollars per Hour

Figure 3.2: Log Wage Density In this context it is useful to transform the data by taking the natural logarithm5 . Figure 3.2 shows the density of log hourly wages log(wage) for the same population, with its mean 2.95 drawn in with the arrow. The density of log wages is much less skewed and fat-tailed than the density of the level of wages, so its mean E (log(wage)) = 2:95 is a much better (more robust) measure6 of central tendency of the distribution. For this reason, wage regressions typically use log wages as a dependent variable rather than the level of wages.

3.3

Conditional Expectation

We saw in Figure 3.2 the density of log wages. Is this wage distribution the same for all workers, or does the wage distribution vary across subpopulations? To answer this question, we can compare wage distributions for di¤erent groups – for example, men and women. The plot on the left in Figure 3.3 displays the densities of log wages for U.S. men and women with their means (3.05 and 2.81) indicated by the arrows. We can see that the two wage densities take similar shapes but the density for men is somewhat shifted to the right with a higher mean. The values 3.05 and 2.81 are the mean log wages in the subpopulations of men and women workers. They are called the conditional means (or conditional expectations) of log wages given gender. We can write their speci…c values as E (log(wage) j gender = man) = 3:05

(3.1)

E (log(wage) j gender = woman) = 2:81:

(3.2)

We call these means conditional as they are conditioning on a …xed value of the variable gender. While you might not think of gender as a random variable, it is random from the viewpoint of 5 6

Throughout the text, we will use log(y) to denote the natural logarithm of y: More precisely, the geometric mean exp (E (log w)) = $19:11 is a robust measure of central tendency.

32

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION

Women

0

1

Log Wage Density

Log Wage Density

white men white women black men black women

Men

2

3

4

5

6

1

Log Dollars per Hour

2

3

4

5

Log Dollars per Hour

Figure 3.3: Left: Log Wage Density for Women and Men. Right: Log Wage Density by Gender and Race econometric analysis. If you randomly select an individual, the gender of the individual is unknown and thus random. (In the population of U.S. workers, the probability that a worker is a woman happens to be 43%.) In observational data, it is most appropriate to view all measurements as random variables, and the means of subpopulations are then conditional means. As the two densities in Figure 3.3 appear similar, a hasty inference might be that there is not a meaningful di¤erence between the wage distributions of men and women. Before jumping to this conclusion let us examine the di¤erences in the distributions of Figure 3.3 more carefully. As we mentioned above, the primary di¤erence between the two densities appears to be their means. This di¤erence equals E (log(wage) j gender = man)

E (log(wage) j gender = woman) = 3:05

= 0:24

2:81 (3.3)

A di¤erence in expected log wages of 0.24 implies an average 24% di¤erence between the wages of men and women, which is quite substantial. (For an explanation of logarithmic and percentage di¤erences see the box on Log Di¤erences below.) Consider further splitting the men and women subpopulations by race, dividing the population into whites, blacks, and other races. We display the log wage density functions of four of these groups on the right in Figure 3.3. Again we see that the primary di¤erence between the four density functions is their central tendency. Focusing on the means of these distributions, Table 3.1 reports the mean log wage for each of the six sub-populations. white black other

men 3.07 2.86 3.03

women 2.82 2.73 2.86

Table 3.1: Mean Log Wages by Sex and Race The entries in Table 3.1 are the conditional means of log(wage) given gender and race. For example E (log(wage) j gender = man; race = white) = 3:07

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION

33

and E (log(wage) j gender = woman; race = black) = 2:73 One bene…t of focusing on conditional means is that they reduce complicated distributions to a single summary measure, and thereby facilitate comparisons across groups. Because of this simplifying property, conditional means are the primary interest of regression analysis and are a major focus in econometrics. Table 3.1 allows us to easily calculate average wage di¤erences between groups. For example, we can see that the wage gap between men and women continues after disaggregation by race, as the average gap between white men and white women is 25%, and that between black men and black women is 13%. We also can see that there is a race gap, as the average wages of blacks are substantially less than the other race categories. In particular, the average wage gap between white men and black men is 21%, and that between white women and black women is 9%.

34

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION Log Di¤erences A useful approximation for the natural logarithm for small x is log (1 + x)

x:

(3.4)

This can be derived from the in…nite series expansion of log (1 + x) : x2 x3 + 2 3 = x + O(x2 ):

log (1 + x) = x

x4 + 4

The symbol O(x2 ) means that the remainder is bounded by Ax2 as x ! 0 for some A < 1: A plot of log (1 + x) and the linear approximation x is shown in the following …gure. We can see that log (1 + x) and the linear approximation x are very close for jxj 0:1, and reasonably close for jxj 0:2, but the di¤erence increases with jxj. 0.4

0.2

-0.4

-0.2

0.2

0.4

x

-0.2

-0.4

log(1 + x) Now, if y is c% greater than y; then y = (1 + c=100)y: Taking natural logarithms, log y = log y + log(1 + c=100) or

c 100 where the approximation is (3.4). This shows that 100 multiplied by the di¤erence in logarithms is approximately the percentage di¤erence between y and y , and this approximation is quite good for jcj 10%: log y

3.4

log y = log(1 + c=100)

Conditional Expectation Function

An important determinant of wage levels is education. In many empirical studies economists measure educational attainment by the number of years of schooling, and we will write this variable as education 7 . 7

Here, education is de…ned as years of schooling beyond kindergarten. A high school graduate has education=12, a college graduate has education=16, a Master’s degree has education=18, and a professional degree (medical, law or

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION

35

The conditional mean of log wages given gender, race, and education is a single number for each category. For example E (log(wage) j gender = man; race = white; education = 12) = 2:84

4.0

We display in Figure 3.4 the conditional means of log(wage) for white men and white women as a function of education. The plot is quite revealing. We see that the conditional mean is increasing in years of education, but at a di¤erent rate for schooling levels above and below nine years. Another striking feature of Figure 3.4 is that the gap between men and women is roughly constant for all education levels. As the variables are measured in logs this implies a constant average percentage gap between men and women regardless of educational attainment.

●

3.5

●

3.0

● ● ●

●

2.5

Log Dollars per Hour

●

●

● ● ●

2.0

●

●

4

6

8

10

12

14

16

white men white women 18

20

Years of Education

Figure 3.4: Mean Log Wage as a Function of Years of Education In many cases it is convenient to simplify the notation by writing variables using single characters, typically y; x and/or z. It is conventional in econometrics to denote the dependent variable (e.g. log(wage)) by the letter y; a conditioning variable (such as gender ) by the letter x; and multiple conditioning variables (such as race, education and gender ) by the subscripted letters x1 ; x2 ; :::; xk . Conditional expectations can be written with the generic notation E (y j x1 ; x2 ; :::; xk ) = m(x1 ; x2 ; :::; xk ): We call this the conditional expectation function (CEF). The CEF is a function of (x1 ; x2 ; :::; xk ) as it varies with the variables. For example, the conditional expectation of y = log(wage) given (x1 ; x2 ) = (gender ; race) is given by the six entries of Table 3.1. The CEF is a function of (gender ; race) as it varies across the entries. For greater compactness, we will typically write the conditioning variables as a vector in Rk : 1 0 x1 B x2 C C B (3.5) x = B . C: @ .. A xk

PhD) has education=20.

36

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION

Here we follow the convention of using lower case bold italics x to denote a vector. Given this notation, the CEF can be compactly written as E (y j x) = m (x) : Sometimes, it is useful to notationally distinguish E (y j x) as the CEF evaluated at the random vector x, from E (y j x = u) as the CEF evaluated at the …xed value u: (And it is mathematically correct to do so.) The …rst expression E (y j x) is a random variable and the second expression E (y j x = u) is a function. We will not always enforce this distinction as it can become notationally burdensome. Hopefully, the use of E (y j x) should be apparent from the context.

3.5

Continuous Variables

4.0

In the previous sections, we implicitly assumed that the conditioning variables are discrete. However, many conditioning variables are continuous. In this section, we take up this case and assume that the variables (y; x) are continuously distributed with a joint density function f (y; x): As an example, take y = log(wage) and x = experience, the number of years of labor market experience. The contours of their joint density are plotted on the left side of Figure 3.5 for the population of white men with 12 years of education.

Log Wage Conditional Density

3.0 2.0

2.5

Log Dollars per Hour

3.5

Exp=5 Exp=10 Exp=25 Exp=40

0

10

20

30

40

50

1.0

Labor Market Experience (Years)

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Log Dollars per Hour

Figure 3.5: Left: Joint density of log(wage) and experience and conditional mean of log(wage) given experience for white men with education=12. Right: Conditional densities of log(wage) for white men with education=12. Given the joint density f (y; x) the variable x has the marginal density Z fx (x) = f (y; x)dy: R

For any x such that fx (x) > 0 the conditional density of y given x is de…ned as fyjx (y j x) =

f (y; x) : fx (x)

(3.6)

The conditional density is a slice of the joint density f (y; x) holding x …xed. We can visualize this by slicing the joint density function at a speci…c value of x parallel with the y-axis. For example,

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION

37

take the density contours on the left side of Figure 3.5 and slice through the contour plot at a speci…c value of experience. This gives us the conditional density of log(wage) for white men with 12 years of education and this level of experience. We do this for four levels of experience (5, 10, 25, and 40 years), and plot these densities on the right side of Figure 3.5. We can see that the distribution of wages shifts to the right and becomes more di¤use as experience increases from 5 to 10 years, and from 10 to 25 years, but there is little change from 25 to 40 years experience. The CEF of y given x is the mean of the conditional density (3.6) Z m (x) = E (y j x) = yfyjx (y j x) dy: (3.7) R

Intuitively, m (x) is the mean of y for the idealized subpopulation where the conditioning variables are …xed at x. This is idealized since x is continuously distributed so this subpopulation is in…nitely small. In Figure 3.5 the CEF of log(wage) given experience is plotted as the solid line. We can see that the CEF is a smooth but nonlinear function. The CEF is initially increasing in experience, ‡attens out around experience = 30, and then decreases for high levels of experience.

3.6

Law of Iterated Expectations

An extremely useful tool from probability theory is the law of iterated expectations. An important special case is the known as the Simple Law.

Theorem 3.6.1 Simple Law of Iterated Expectations If E jyj < 1 then for any random vector x, E (E (y j x)) = E (y)

The simple law states that the expectation of the conditional expectation is the unconditional expectation. In other words, the average of the conditional averages is the unconditional average. When x is discrete 1 X E (E (y j x)) = E (y j x j ) Pr (x = x j ) j=1

and when x is continuous E (E (y j x)) =

Z

Rk

E (y j x ) fx (x)dx:

Going back to our investigation of average log wages for men and women, the simple law states that E (log(wage) j gender = man) Pr (gender = man)

+E (log(wage) j gender = woman) Pr (gender = woman)

= E (log(wage)) : Or numerically,

3:05 0:57 + 2:79 0:43 = 2:92: The general law of iterated expectations allows two sets of conditioning variables.

38

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION Theorem 3.6.2 Law of Iterated Expectations If E jyj < 1 then for any random vectors x1 and x2 , E (E (y j x1 ; x2 ) j x1 ) = E (y j x1 )

Notice the way the law is applied. The inner expectation conditions on x1 and x2 , while the outer expectation conditions only on x1 : The iterated expectation yields the simple answer E (y j x1 ) ; the expectation conditional on x1 alone. Sometimes we phrase this as: “The smaller information set wins.” As an example E (log(wage) j gender = man; race = white) Pr (race = whitejgender = man)

+E (log(wage) j gender = man; race = black) Pr (race = blackjgender = man)

+E (log(wage) j gender = man; race = other) Pr (race = otherjgender = man)

= E (log(wage) j gender = man) or numerically

3:07 0:84 + 2:86 0:08 + 3:05 0:08 = 3:05 A property of conditional expectations is that when you condition on a random vector x you can e¤ectively treat it as if it is constant. For example, E (x j x) = x and E (g (x) j x) = g (x) for any function g( ): The general property is known as the conditioning theorem.

Theorem 3.6.3 Conditioning Theorem If E jg (x) yj < 1

(3.8)

then E (g (x) y j x) = g (x) E (y j x)

(3.9)

E (g (x) y) = E (g (x) E (y j x))

(3.10)

and

The proofs of Theorems 3.6.1, 3.6.2 and 3.6.3 are given in Section 3.30.

3.7

Monotonicity of Conditioning

What is the e¤ect of increasing the amount of information when constructing a conditional expectation? That is, how do we compare E (y j x1 ) versus E (y j x1 ; x2 )? We have seen that by increasing the conditioning set, the conditional expectation reveals greater detail about the distribution of y: Is there something more that can be said? It turns out that there is a simple relationship induced by conditioning. We can think of the conditional mean E (y j x1 ) as the “explained portion” of y: The remainder y E (y j x1 ) is the “unexplained portion”. The simple relationship we now derive shows that the variance of this unexplained portion decreases when we condition on more variables. This relationship is monotonic in the sense that increasing the amont of information always decreases the variance of the unexplained portion.

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION

39

Theorem 3.7.1 If Ey 2 < 1 then var (y)

E (y j x1 ))

var (y

var (y

E (y j x1 ; x2 ))

Theorem 3.7.1 says that the variance of the di¤erence between y and its conditional mean (weakly) decreases whenever an additional variable is added to the conditioning information. The proof of Theorem 3.7.1 is given in Section 3.30.

3.8

CEF Error

The CEF error e is de…ned as the di¤erence between y and the CEF evaluated at the random vector x: e = y m(x): By construction, this yields the formula y = m(x) + e:

(3.11)

In (3.11) it is useful to understand that the error e is derived from the joint distribution of (y; x); and so its properties are derived from this construction. A key property of the CEF error is that it has a conditional mean of zero. To see this, by the linearity of expectations, the de…nition m(x) = E (y j x) and the Conditioning Theorem E (e j x) = E ((y

m(x)) j x)

= E (y j x)

= m(x)

E (m(x) j x)

m(x)

= 0: This fact can be combined with the law of iterated expectations to show that the unconditional mean is also zero. E (e) = E (E (e j x)) = E (0) = 0 We state this and some other results formally. Theorem 3.8.1 Properties of the CEF error If E jyj < 1 then 1. E (e j x) = 0: 2. E (e) = 0: 3. If E jyjr < 1 for r

1 then E jejr < 1:

4. For any function h (x) such that E jh (x) ej < 1 then E (h (x) e) = 0 The proof of the third result is deferred to Section 3.30: The fourth result, whose proof is left to Exercise 3.3, says that e is uncorrelated with any function of the regressors. The equations y = m(x) + e E (e j x) = 0:

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION

40

e

−1.0

−0.5

0.0

0.5

1.0

together imply that m(x) is the CEF of y given x. It is important to understand that this is not a restriction. These equations hold true by de…nition. The condition E (e j x) = 0 is implied by the de…nition of e as the di¤erence between y and the CEF m (x) : The equation E (e j x) = 0 is sometimes called a conditional mean restriction, since the conditional mean of the error e is restricted to equal zero. The property is also sometimes called mean independence, for the conditional mean of e is 0 and thus independent of x. However, it does not imply that the distribution of e is independent of x: Sometimes the assumption “e is independent of x” is added as a convenient simpli…cation, but it is not generic feature of the conditional mean. Typically and generally, e and x are jointly dependent, even though the conditional mean of e is zero. As an example, the contours of the joint density of e and experience are plotted in Figure 3.6 for the same population as Figure 3.5. The error e has a conditional mean of zero for all values of experience, but the shape of the conditional distribution varies with the level of experience.

0

10

20

30

40

50

Labor Market Experience (Years)

Figure 3.6: Joint density of CEF error e and experience for white men with education=12. As a simple example of a case where x and e are mean independent yet dependent, let y = xu where x and u are independent and Eu = 1: Then E (y j x) = E (xu j x) = xE (u j x) = x so the CEF equation is y =x+e where e = x(u

1):

Note that even though e is not independent of x; E (e j x) = E (x(u

1) j x) = xE ((u

1) j x) = 0

and is thus mean independent. An important measure of the dispersion about the CEF function is the unconditional variance of the CEF error e: We write this as 2

= var (e) = E (e

Ee)2 = E e2 :

Theorem 3.8.1.3 implies the following simple but useful result.

41

CHAPTER 3. CONDITIONAL EXPECTATION AND PROJECTION Theorem 3.8.2 If Ey 2 < 1 then

3.9

2

0 is a pre-speci…ed constant. This con…dence interval is symmetric about the point estimate b; and its length is proportional to the standard error s(b):

130

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES Equivalently, Cn is the set of parameter values for absolute value) than c; that is ( Cn = f : jtn ( )j

cg =

such that the t-statistic tn ( ) is smaller (in

:

c

The coverage probability of this con…dence interval is Pr ( 2 Cn ) = Pr (jtn ( )j

b

s(b)

)

c :

c)

which is generally unknown. We can approximate the coverage probability by taking the asymptotic limit as n ! 1: Since jtn ( )j is asymptotically jZj (Theorem 6.11.1), it follows that as n ! 1 that Pr ( 2 Cn ) ! Pr (jZj

c) = (c)

where (u) is given in (6.40). We call this the asymptotic coverage probability. Since the tratio is asymptotically pivotal, the asymptotic coverage probability is independent of the parameter ; and is only a function of c: As we mentioned before, an ideal con…dence interval has a pre-speci…ed probability coverage 1 ; typically 90% or 95%. This means selecting the constant c so that (c) = 1

:

E¤ectively, this makes c a function of ; and can be backed out of a normal distribution table. For example, = 0:05 (a 95% interval) implies c = 1:96 and = 0:1 (a 90% interval) implies c = 1:645: Rounding 1.96 to 2, we obtain the most commonly used con…dence interval in applied econometric practice i h (6.42) Cn = b 2s(b); b + 2s(b) : This is a useful rule-of thumb. This asymptotic 95% con…dence interval Cn is simple to compute and can be roughly calculated from tables of coe¢cient estimates and standard errors. (Technically, it is an asymptotic 95.4% interval, due to the substitution of 2.0 for 1.96, but this distinction is meaningless.)

Theorem 6.12.1 Under Assumption 1.5.1, Assumption 6.3.1, and Assumption 6.9.1, for Cn de…oned in (6.41), Pr ( 2 Cn ) ! (c): For c = 1:96; Pr ( 2 Cn ) ! 0:95:

Con…dence intervals are a simple yet e¤ective tool to assess estimation uncertainty. When reading a set of empirical results, look at the estimated coe¢cient estimates and the standard errors. For a parameter of interest, compute the con…dence interval Cn and consider the meaning of the spread of the suggested values. If the range of values in the con…dence interval are too wide to learn about ; then do not jump to a conclusion about based on the point estimate alone.

6.13

Regression Intervals

In the linear regression model the conditional mean of yi given xi = x is m(x) = E (yi j xi = x) = x0 :

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

131

Figure 6.7: Wage on Education Regression Intervals In some cases, we want to estimate m(x) at a particular point x: Notice that this is a (linear) 0 function b = b = x0 b and H = x; so qof : Letting h( ) = x and = h( ); we see that m(x) s(b) = n 1 x0 Vb x: Thus an asymptotic 95% con…dence interval for m(x) is 0b

x

q 2 n

1 x0 V b

x :

It is interesting to observe that if this is viewed as a function of x; the width of the con…dence set is dependent on x: To illustrate, we return to the log wage regression (4.9) of Section 4.4. The estimated regression equation is \ log(W age) = x0 b = 0:626 + 0:156x:

where x = dducation. The White covariance matrix estimate is Vb b =

7:092 0:445

0:445 0:029

and the sample size is n = 61: Thus the 95% con…dence interval for the regression takes the form r 1 (7:092 0:89x + 0:029x2 ) : 0:626 + 0:156x 2 61 The estimated regression and 95% intervals are shown in Figure 6.7. Notice that the con…dence bands take a hyperbolic shape. This means that the regression line is less precisely estimated for very large and very small values of education. Plots of the estimated regression line and con…dence intervals are especially useful when the regression includes nonlinear terms. To illustrate, consider the log wage regression (4.10) which includes experience and its square. \ log(W age) = 1:06 + 0:116 education + 0:010 experience

0:014 experience2 =100

(6.43)

and has n = 2454 observations. We are interested in plotting the regression estimate and regression intervals as a function of experience. Since the regression also includes education, to plot the

132

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

Figure 6.8: Wage on Experience Regression Intervals estimates in a simple graph we need to …x education at a speci…c value. We select education=12. This only a¤ects the level of the estimated regression, since education enters without an interaction. De…ne the points of evaluation 1 0 1 B 12 C C x=B A @ x 2 x =100 where x =experience. The covariance matrix estimate is 0 22:92 1:0601 0:56687 B 1:0601 0:06454 :0080737 Vb b = B @ 0:56687 :0080737 :040736 0:86626 :0066749 :075583

1 0:86626 :0066749 C C: :075583 A 0:14994

Thus the regression interval for education=12, as a function of x =experience is 1:06 + 0:116 12 + 0:010 x 0:014 x2 =100 v 0 u 22:92 u u B 1u 1 1:0601 u 1 12 x x2 =100 B @ t 0:56687 50 2454 0:86626

= 2:452 + 0:010 x :00014 x2 2 p 27:592 3:8304 x + 0:23007 x2 100

1:0601 0:06454 :0080737 :0066749

0:56687 :0080737 :040736 :075583

10 1 0:86626 B 12 :0066749 C CB x :075583 A @ x2 =100 0:14994

0:00616 x3 + 0:0000611 x4

The estimated regression and 95% intervals are shown in Figure 6.8. The regression interval widens greatly for small and large values of experience, indicating considerable uncertainty about the e¤ect of experience on mean wages for this population. The con…dence bands take a more complicated shape than in Figure 6.7 due to the nonlinear speci…cation.

1 C C A

133

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

6.14

Forecast Intervals

For a given value of xi = x; we may want to forecast (guess) yi out-of-sample. A reasonable rule is the conditional mean m(x) as it is the mean-square-minimizing forecast. A point forecast is the estimated conditional mean m(x) ^ = x0 b . We would also like a measure of uncertainty for the forecast. The forecast error is e^i = yi m(x) ^ = ei x0 b : As the out-of-sample error ei is independent of the in-sample estimate ^ ; this has variance b

E^ e2i = E e2i j xi = x + x0 E b 2

=

(x) + n

1 0

x V x:

0

x

the natural estimate of this variance is ^ 2 + n 1 x0 Vb x; so a standard q error for the forecast is s^(x) = ^ 2 + n 1 x0 Vb x: Notice that this is di¤erent from the standard error for the conditional mean. If we have an estimate of the conditional variance function, e.g. q 0 2 2 ~ (x) = e z from (9.5), then the forecast standard error is s^(x) = ~ (x) + n 1 x0 Vb x

Assuming E e2i j xi =

2;

It would appear natural to conclude that an asymptotic 95% forecast interval for yi is h i x0 b 2^ s(x) ;

but this turns out to be incorrect. In general, the validity of an asymptotic con…dence interval is based on the asymptotic normality of the studentized ratio. In the present case, this would require the asymptotic normality of the ratio x0 b

ei

:

s^(x)

But no such asymptotic approximation can be made. The only special exception is the case where ei has the exact distribution N(0; 2 ); which is generally invalid. To get an accurate forecast interval, we need to estimate the conditional distribution of ei given xi = x; which is a much more di¢culth task. Perhaps i due to this di¢culty, many applied forecasters 0 b use the simple approximate interval x 2^ s(x) despite the lack of a convincing justi…cation.

6.15

Wald Statistic

Let = h( ) : Rk ! Rq be any parameter vector of interest, b its estimate and Vb covariance matrix estimator. Consider the quadratic form 0

Wn ( ) = n b

Vb

1

b

:

its

(6.44)

When q = 1; then Wn ( ) = tn ( )2 is the square of the t-ratio. When q > 1; Wn ( ) is typically called a Wald statistic. We are interested in its sampling distribution. The asymptotic distribution of Wn ( ) is simple to derive given Theorem 6.9.2 and Theorem 6.9.3, which show that p d ! Z N (0; V ) n b

and

It follows that Wn ( ) =

p

n b

Vb

0

p

!V :

Vb

1p

n b

d

! Z0 V

1

Z

(6.45)

134

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

a quadratic in the normal random vector Z: Here we can appeal to a useful result from probability theory. (See Theorem B.9.3 in the Appendix.)

q; then Z 0 A

Theorem 6.15.1 If Z N (0; A) with A > 0; q random variable with q degrees of freedom.

1

Z

2; q

a chi-square

The asymptotic distribution in (6.45) takes exactly this form. Note that V > 0 since H is full rank under Assumption 6.9.1 It follows that Wn ( ) converges in distribution to a chi-square random variable. Theorem 6.15.2 Under Assumption 1.5.1, Assumption 6.3.1, and Assumption 6.9.1, as n ! 1; d Wn ( ) ! 2q :

Theorem 6.15.2 is used to justify multivariate con…dence regions and mutivariate hypothesis tests.

6.16

Con…dence Regions

A con…dence region Cn is a set estimator for 2 Rq when q > 1: A con…dence region Cn is a set in Rq intended to cover the true parameter value with a pre-selected probability 1 : Thus an ideal con…dence region has the coverage probability Pr( 2 Cn ) = 1 . In practice it is typically not possible to construct a region with exact coverage, but we can calculate its asymptotic coverage. When the parameter estimate satis…es the conditions of Theorem 6.15.2, a good choice for a con…dence region is the ellipse Cn = f : Wn ( ) c1 g :

with c1 the 1 ’th quantile of the 2q distribution. (Thus Fq (c1 can be found from the 2q critical value table. Theorem 6.15.2 implies 2 q

Pr ( 2 Cn ) ! Pr

c1

)=1

:) These quantiles

=1

which shows that Cn has asymptotic coverage (1 )%: To illustrate the construction of a con…dence region, consider the estimated regression (6.43) of the model \ log(W age) =

+

1

education +

2

experience +

3

experience2 =100:

Suppose that the two parameters of interest are the percentage return to education 1 = 100 1 and the percentage return to experience for individuals with 10 years experience 2 = 100 2 + 20 3 . (We need to condition on the level of experience since the regression is quadratic in experience.) These two parameters are a linear transformation of the regression parameters with point estimates b=

0 100 0 0 0 0 100 20

and have the covariance matrix estimate

b=

11:6 0:72

;

135

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

Figure 6.9: Con…dence Region for Return to Experience and Return to Education

Vb

=

0 100 0 0 0 0 100 20

=

645:4 67:387 67:387 165

with inverse

Thus the Wald statistic is Wn ( ) = n b

= 2454

1

Vb 0

Vb

11:6 0:72

= 3:97 (11:6

0:0016184 0:00066098

=

1

1 2 2

1)

b

0

1 0 0 B 100 0 C C Vb b B @ 0 100 A 0 20 0

0:00066098 0:0063306

0:0016184 0:00066098

3:2441 (11:6

:

0:00066098 0:0063306

1 ) (0:72

2)

11:6 0:72

+ 15:535 (0:72

1 2 2 2)

The 90% quantile of the 22 distribution is 4.605 (we use the 22 distribution as the dimension of is two), so an asymptotic 90% con…dence region for the two parameters is the interior of the ellipse 3:97 (11:6

2 1)

3:2441 (11:6

1 ) (0:72

2)

+ 15:535 (0:72

2 2)

= 4:605

which is displayed in Figure 6.9. Since the estimated correlation of the two coe¢cient estimates is small (about 0.2) the ellipse is close to circular.

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

6.17

136

Semiparametric E¢ciency in the Projection Model

In Section 5.4 we presented the Gauss-Markov theorem, which stated that in the homoskedastic CEF model, in the class of linear unbiased estimators the one with the smallest variance is leastsquares. As we noted in that section, the restriction to linear unbiased estimators is unsatisfactory as it leaves open the possibility that an alternative (non-linear) estimator could have a smaller asymptotic variance. In addition, the restriction to the homoskedastic CEF model is also unsatisfactory as the projection model is more relevant for empirical application. The question remains: what is the most e¢cient estimator of the projection coe¢cient (or functions = h( )) in the projection model? It turns out that it is straightforward to show that the projection model falls in the estimator class considered in Proposition 2.13.2. It follows that the least-squares estimator is semiparametrically e¢cient in the sense that it has the smallest asymptotic variance in the class of semiparametric estimators of . This is a more powerful and interesting result than the Gauss-Markov theorem. To see this, it is worth rephrasing Proposition 2.13.2 with amended notation. Suppose that P a parameter of interest is = g( n) where = Ez i ; for which the moment estimators are b = n1 ni=1 o zi 2 and b = g(b ): Let L2 (g) = F : E kzk < 1; g (u) is continuously di¤erentiable at u = Ez be the set of distributions for which b satis…es the central limit theorem.

Proposition 6.17.1 In the class of distributions F 2 L2 (g); b is semiparametrically e¢cient for in the sense that its asymptotic variance equals the semiparametric e¢ciency bound.

Proposition 6.17.1 says that under the minimal conditions in which b is asymptotically normal, then no semiparametric estimator can have a smaller asymptotic variance than b. To show that an estimator is semiparametrically e¢cient it is su¢cient to show that it falls in the class covered by this Proposition. To show that the projection model falls in this class, we write = Qxx1 Qxy = g ( ) where = Ez i and z i = (xi x0i ; xi yi ) : The class L2 (g) equals the class of distributions n o L4 ( ) = F : Ey 4 < 1; E kxk4 < 1; Exi x0i > 0 : Proposition 6.17.2 In the class of distributions F 2 L4 ( ); the leastsquares estimator b is semiparametrically e¢cient for . The least-squares estimator is an asymptotically e¢cient estimator of the projection coe¢cient because the latter is a smooth function of sample moments and the model implies no further restrictions. However, if the class of permissible distributions is restricted to a strict subset of L4 ( ) then least-squares can be ine¢cient. For example, the linear CEF model with heteroskedastic errors is a strict subset of L4 ( ); and the GLS estimator has a smaller asymptotic variance than OLS. In this case, the knowledge that true conditional mean is linear allows for more e¢cient estimation of the unknown parameter. From Proposition 6.17.1 we can also deduce that plug-in estimators b = h( b ) are semiparametrically e¢cient estimators of = h( ) when h is continuously di¤erentiable. We can also deduce

137

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES that other parameters estimators are semiparametrically e¢cient, such as ^ 2 for note that we can write 2

= E yi

2:

To see this,

2

x0i

= Eyi2

2E yi x0i

= Qyy

Qyx Qxx1 Qxy

+

0

E xi x0i

which is a smooth function of the moments Qyy ; Qyx and Qxx : Similarly the estimator ^ 2 equals n

^

2

=

1X 2 e^i n i=1

b yy = Q

b b yx Q b 1Q Q xx xy

Since the variables yi2 ; yi x0i and xi x0i all have …nite variances when F 2 L4 ( ); the conditions of Proposition 6.17.1 are satis…ed. We conclude:

Proposition 6.17.3 In the class of distributions F 2 L4 ( ); ^ 2 is semiparametrically e¢cient for 2 .

6.18

Semiparametric E¢ciency in the Homoskedastic Regression Model*

In Section 6.17 we showed that the OLS estimator is semiparametrically e¢cient in the projection model. What if we restrict attention to the classical homoskedastic regression model? Is OLS still e¢cient in this class? In this section we derive the asymptotic semiparametric e¢ciency bound for this model, and show that it is the same as that obtained by the OLS estimator. Therefore it turns out that least-squares is e¢cient in this class as well. Recall that in the homoskedastic regression model the asymptotic variance of the OLS estimator b for is V 0 = Q 1 2 : Therefore, as described in Section 2.13, it is su¢cient to …nd a parametric xx submodel whose Cramer-Rao bound for estimation of is V 0 : This would establish that V 0 is the semiparametric variance bound and the OLS estimator b is semiparametrically e¢cient for : Let the joint density of y and x be written as f (y; x) = f1 (y j x) f2 (x) ; the product of the conditional density of y given x and the marginal density of x. Now consider the parametric submodel f (y; x j ) = f1 (y j x) 1 + y x0 x0 = 2 f2 (x) : (6.46) You can check that in this submodel the marginal density of x is f2 (x) and the conditional density of y given x is f1 (y j x) 1 + (y x0 ) (x0 ) = 2 : To see that the latter is a valid conditional R density, observe that the regression assumption implies that yf1 (y j x) dy = x0 and therefore Z Z Z 0 0 2 f1 (y j x) 1 + y x x = dy = f1 (y j x) dy + f1 (y j x) y x0 dy x0 = 2 = 1:

138

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES In this parametric submodel the conditional mean of y given x is Z E (y j x) = yf1 (y j x) 1 + y x0 x0 = 2 dy Z Z = yf1 (y j x) dy + yf1 (y j x) y x0 x0 Z Z 2 y x0 f1 (y j x) x0 = yf1 (y j x) dy + Z + y x0 f1 (y j x) dy x0 x0 = 2

=

2

=

2

dy

dy

= x0 ( + ) ;

R using the homoskedasticity assumption (y x0 )2 f1 (y j x) dy = 2 : This means that in this parametric submodel, the conditional mean is linear in x and the regression coe¢cient is ( ) = + : We now calculate the score for estimation of : Since @ @ log f (y; x j ) = log 1 + y @ @

x0

x0

=

2

=

x (y x0 ) = 2 1 + (y x0 ) (x0 ) =

2

the score is

@ log f (y; x j 0 ) = xe= 2 : @ The Cramer-Rao bound for estimation of (and therefore ( ) as well) is s=

E ss0

1

=

4

E (xe) (xe)0

1

=

2

Qxx1 = V 0 :

We have shown that there is a parametric submodel (6.46) whose Cramer-Rao bound for estimation of is identical to the asymptotic variance of the least-squares estimator, which therefore is the semiparametric variance bound. Theorem 6.18.1 In the homoskedastic regression model, the semiparametric variance bound for estimation of is V 0 = 2 Qxx1 and the OLS estimator is semiparametrically e¢cient.

This result is similar to the Gauss-Markov theorem, in that it asserts the e¢ciency of the leastsquares estimator in the context of the homoskedastic regression model. The di¤erence is that the Gauss-Markov theorem states that OLS has the smallest variance among the set of unbiased linear estimators, while Theorem 6.18.1 states that OLS has the smallest asymptotic variance among all regular estimators. This is a much more powerful statement.

6.19

Uniformly Consistent Residuals*

It seems natural to view the residuals e^i as estimates of the unknown errors ei : Are they consistent estimates? In this section we develop an appropriate convergence result. This is not a widely-used technique, and can safely be skipped by most readers. Notice that we can write the residual as e^i = yi

x0i b

= ei + x0i = ei

x0i b

x0i b

:

(6.47)

139

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES p

Since b ! 0 it seems reasonable to guess that e^i will be close to ei if n is large. We can bound the di¤erence in (6.47) using the Schwarz inequality (A.10) to …nd j^ ei

ei j = x0i b

kxi k b

:

(6.48)

To bound (6.48) we can use b = Op (n 1=2 ) from Theorem 6.3.2, but we also need to bound the random variable kxi k. The key is Theorem 2.12.1 which shows that E kxi kr < 1 implies xi = op n1=r uniformly in i; or p n 1=r max kxi k ! 0: 1 i n

Applied to (6.48) we obtain max j^ ei

max kxi k b

ei j

1 i n

1 i n

1=2+1=r

= op (n

):

We have shown the following. Theorem 6.19.1 Under Assumptions 1.5.1, 6.3.1, and E kxi kr < 1, then uniformly in 1 i n e^i = ei + op (n 1=2+1=r ): (6.49)

The rate of convergence in (6.49) depends on r: Assumption 6.3.1 requires r 4; so the slowest rate of convergence is op (n 1=4 ): As r increases, the rate becomes close to Op (n 1=2 ). If the regressor is bounded, kxi k B < 1, then e^i = ei + op (n 1=2 ):

6.20

Asymptotic Leverage*

Recall the de…nition of leverage from (4.21) 1

hii = x0i X 0 X

xi :

These are the diagonal elements of the projection matrix P and appear in the formula for leaveone-out prediction errors and several covariance matrix estimators. We can show that under iid sampling the leverage values are uniformly asymptotically small. Let min (A) and max (A) denote the smallest and largest eigenvalues of a symmetric square matrix A; and note that max (A 1 ) = ( min (A)) 1 : p p Since n1 X 0 X ! Qxx > 0 then by the CMT, min n1 X 0 X ! min (Qxx ) > 0: (The latter is positive since Qxx is positive de…nite and thus all its eigenvalues are positive.) Then by the Trace Inequality (A.13) hii = x0i X 0 X

xi

1 0 XX n

= tr

max

=

1

1 0 XX n 1 0 XX n

1

1 xi x0i n !

!

1

tr

1 xi x0i n

1

1 kxi k2 n 1 ( min (Qxx ) + op (1)) 1 max kxi k2 : n1 i n min

(6.50)

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES Theorem 2.12.1 shows that E kxi kr < 1 implies max1 op n2=r 1 .

i n kxi k

140

= op n1=r and thus (6.50) is

Theorem 6.20.1 Under Assumption 1.5.1 and E kxi kr < 1 for some r 2; then uniformly in 1 i n, hii = op n2=r 1 :

For any r 2 then hii = op (1) (uniformly in i n): Larger r implies a stronger rate of convergence, for example r = 4 implies hii = op n 1=2 : Theorem (6.20.1) implies that under random sampling with …nite variances and large samples, no individual observation should have a large leverage value. Consequently individual observations should not be in‡uential, unless one of these conditions is violated.

141

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

Exercises Exercise 6.1 Take the model yi = x01i 1 +x02i 2 +ei with Exi ei = 0: Suppose that 1 is estimated by regressing yi on x1i only. Find the probability limit of this estimator. In general, is it consistent for 1 ? If not, under what conditions is this estimator consistent for 1 ? Exercise 6.2 Let y be n regression estimator

where

1; X be n b=

k (rank k): y = X + e with E(xi ei ) = 0: De…ne the ridge

n X

xi x0i + I k

i=1

!

1

n X

xi y i

i=1

!

(6.51)

> 0 is a …xed constant. Find the probability limit of b as n ! 1: Is b consistent for

?

= cn where c > 0 is …xed as n ! 1:

Exercise 6.3 For the ridge regression estimator (6.51), set Find the probability limit of b as n ! 1:

Exercise 6.4 Verify some of the calculations reported in Section 6.4. Speci…cally, suppose that x1i and x2i only take the values f 1; +1g; symmetrically, with Pr (x1i = x2i = 1) = Pr (x1i = x2i = Pr (x1i = 1; x2i =

1) = Pr (x1i = 5 E e2i j x1i = x2i = 4 1 2 : E ei j x1i 6= x2i = 4

1; x2i = 1) = 1=8

Verify the following: 1. Ex1i = 0 2. Ex21i = 1 3. Ex1i x2i =

1 2

4. E e2i = 1 5. E x21i e2i = 1 7 6. E x1i x2i e2i = : 8 Exercise 6.5 Show (6.19)-(6.22). Exercise 6.6 The model is yi = x0i + ei E (xi ei ) = 0 = E xi x0i e2i : Find the method of moments estimators ( b ; b ) for ( ; ) :

(a) In this model, are ( b ; b ) e¢cient estimators of ( ; )?

(b) If so, in what sense are they e¢cient?

1) = 3=8

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

142

Exercise 6.7 Of the variables (yi ; yi ; xi ) only the pair (yi ; xi ) are observed. In this case, we say that yi is a latent variable. Suppose yi

= x0i + ei

E (xi ei ) = 0 yi = yi + u i where ui is a measurement error satisfying E (xi ui ) = 0 E (yi ui ) = 0 Let b denote the OLS coe¢cient from the regression of yi on xi : (a) Is

the coe¢cient from the linear projection of yi on xi ?

(b) Is b consistent for

as n ! 1?

(c) Find the asymptotic distribution of

p

n b

as n ! 1:

Exercise 6.8 Find the asymptotic distribution of

p

n ^2

2

as n ! 1:

Exercise 6.9 The model is yi = xi + ei E (ei j xi ) = 0 where xi 2 R: Consider the two estimators b =

e =

Pn xi yi Pi=1 n 2 i=1 xi n 1 X yi : n xi i=1

(a) Under the stated assumptions, are both estimators consistent for ? (b) Are there conditions under which either estimator is e¢cient? Exercise 6.10 In the homoskedastic regression model y = X + e with E(ei j xi ) = 0 and E(e2i j xi ) = 2 ; suppose ^ is the OLS estimate of with covariance matrix V^ ; based on a sample of size n: Let ^ 2 be the estimate of 2 : You wish to forecast an out-of-sample value of yn+1 given that xn+1 = x: Thus the available information is the sample (y; X); the estimates ( b ; Vb ; ^ 2 ), the residuals e ^; and the out-of-sample value of the regressors, xn+1 : (a) Find a point forecast of yn+1 : (b) Find an estimate of the variance of this forecast.

Chapter 7

Restricted Estimation 7.1

Introduction

In the linear projection model yi = x0i + ei E (xi ei ) = 0 a common task is to impose a constraint on the coe¢cient vector . For example, partitioning 0 0 x0i = (x01i ; x02i ) and 0 = 1 ; 2 ; a typical constraint is an exclusion restriction of the form 2 = 0: In this case the constrained model is yi = x01i

1

+ ei

E (xi ei ) = 0 At …rst glance this appears the same as the linear projection model, but there is one important di¤erence: the error ei is uncorrelated with the entire regressor vector x0i = (x01i ; x02i ) not just the included regressor x1i : In general, a set of q linear constraints on takes the form R0

=c

(7.1)

where R is k q; rank(R) = q < k and c is q 1: The assumption that R is full rank means that the constraints are linearly independent (there are no redundant or contradictory constraints). The constraint 2 = 0 discussed above is a special case of the constraint (7.1) with R=

0 I

;

(7.2)

a selector matrix, and c = 0. Another common restriction is that a set of coe¢cients sum to a known constant, i.e. 1 + 2 = 1: This constraint arises in a constant-return-to-scale production function. Other common restrictions include the equality of coe¢cients 1 = 2 ; and equal and o¤setting coe¢cients 1 = 2: A typical reason to impose a constraint is that we believe (or have information) that the constraint is true. By imposing the constraint we hope to improve estimation e¢ciency. The goal is to obtain consistent estimates with reduced variance relative to the unconstrained estimator. The questions then arise: How should we estimate the coe¢cient vector imposing the linear restriction (7.1)? If we impose such constraints, what is the sampling distribution of the resulting estimator? How should we calculate standard errors? These are the questions explored in this chapter. 143

144

CHAPTER 7. RESTRICTED ESTIMATION

7.2

Constrained Least Squares

An intuitively appealing method to estimate a constrained linear projection is to minimize the least-squares criterion subject to the constraint R0 = c. This estimator is e = argmin SSEn ( )

(7.3)

R0 =c

where SSEn ( ) =

n X

x0i

yi

2

= y0y

2y 0 X +

0

X 0X :

i=1

The estimator e minimizes the sum of squared errors over all such that the restriction (7.1) e holds. We call the constrained least-squares (CLS) estimator. We follow the convention of using a tilde “~” rather than a hat “^” to indicate that e is a restricted estimator in contrast to the unrestricted least-squares estimator b ; and write it as e cls when we want to be clear that the estimation method is CLS. One method to …nd the solution to (7.3) uses the technique of Lagrange multipliers. The problem (7.3) is equivalent to the minimization of the Lagrangian 1 L( ; ) = SSEn ( ) + 2 over ( ; ); where is an s minimization of (7.4) are

R0

c

(7.4)

1 vector of Lagrange multipliers. The …rst-order conditions for

@ L( e ; e ) = @

and

0

Premultiplying (7.5) by R0 (X 0 X)

X 0y + X 0X e + R e = 0

@ L( ; ) = R0 e @

1

c = 0:

(7.6)

we obtain

R0 b + R0 e + R0 X 0 X

1

(7.5)

1

Re = 0

where b = (X 0 X) X 0 y is the unrestricted least-squares estimator. Imposing R0 e (7.6) and solving for e we …nd h i 1 e = R0 X 0 X 1 R R0 b c :

(7.7) c = 0 from

Substuting this expression into (7.5) and solving for e we …nd the solution to the constrained minimization problem (7.3) h i 1 1 1 e =b X 0X R R0 X 0 X R R0 b c : (7.8) cls This is a general formula for the CLS estimator. It also can be written as h i 1 1 0b 1 e =b Q b R R Q R R0 b c : cls xx xx

Given e cls the residuals are

The moment estimator of

2

e~i = yi

is

x0i e cls : n

~ 2cls =

1X 2 e~i : n i=1

145

CHAPTER 7. RESTRICTED ESTIMATION

7.3

Exclusion Restriction

While (7.8) is a general formula for the CLS estimator, in most cases the estimator can be found by applying least-squares to a reparameterized equation. To illustrate, let us return to the …rst example presented at the beginning of the chapter – a simple exclusion restriction. Recall the unconstrained model is (7.9) yi = x01i 1 + x02i 2 + ei the exclusion restriction is

2

= 0; and the constrained equation is yi = x01i

+ ei :

1

(7.10)

In this setting the CLS estimator is OLS of yi on x1i : (See Exercise 7.1.) We can write this as ! 1 n ! n X X e = x1i yi : x1i x01i (7.11) 1 i=1

i=1

The CLS estimator of the entire vector

0

0 1;

=

e=

e

0 2

is

1

:

0

(7.12)

It is not immediately obvious, but (7.8) and (7.12) are algebraically (and numerically) equivalent. To see this, the …rst component of (7.8) with (7.2) is # " 1 1 1 0 0 e = I 0 b Q b b 0 I b : 0 I Q 1 xx xx I I Using (4.31) this equals e

1

= b1 =

=

b 12 Q b 22 Q

1

b

2

b +Q b b 1 Q b b 1b 1 11 2 12 Q22 Q22 1 2 b 1Q b 1 Q b 1y Q b 12 Q b 2y Q 22 11 2

b 1y b 1 Q = Q 11 2

b 11 b 1 Q = Q 11 2 b 1Q b 1y = Q 11

b 1 b b 1 Q b b 1b +Q 11 2 12 Q22 Q22 1 Q22 1 Q2y

b 12 Q b 1Q b 21 Q b 1Q b 1y Q 22 11

b 1Q b 21 Q b 1y Q 11

b 1Q b 12 Q b 21 Q b 1Q b 1y Q 22 11

which is (7.12) as originally claimed.

7.4

Minimum Distance

A minimum distance estimator tries to …nd a parameter value which satis…es the constraint which is as close as possible to the unconstrained estimate. Let b be the unconstrained leastsquares estimator, and for some k k positive de…nite weight matricx W n > 0 de…ne Jn ( ) = n b

0

Wn b

(7.13)

= argmin Jn ( ) :

(7.14)

This is a (squared) weighted Euclidean distance between b and : Jn ( ) is small if is close to b , and is minimized at zero only if = b . A minimum distance estimator e for e minimizes Jn ( ) subject to the constraint (7.1), that is, e

md

R0 =c

146

CHAPTER 7. RESTRICTED ESTIMATION 0 Vb

1

: To see this, rewrite the leastsquares criterion as follows. Write the unconstrained least-squares …tted equation as yi = x0i b + e^i and substitute this equation into SSEn ( ) to obtain The CLS estimator is the special case when W n =

SSEn ( ) =

n X

yi

i=1

x0i b + e^i

x0i

2

i=1

= =

n X

n X i=1 2

e^2i + b

2

x0i 0

= s (n

n X

xi x0i

i=1

!

b

k + Jn ( )) (7.15) Pn where the third equality uses the fact that i=1 xi e^i = 0; and the last line holds when W n = 1 0 1 Pn = Vb xi x0i s 2 : The expression (7.15) only depends on through Jn ( ) : Thus n i=1 1 0 : minimization of SSEn ( ) and Jn ( ) are equivalent, and hence e md = e cls when W n = Vb We can solve for e explicitly by the method of Lagrange multipliers. The Lagrangian is md

1 L( ; ) = Jn ( ; W n ) + 2

0

R0

c

which is minimized over ( ; ): The solution is e

md

=b

W n 1 R R0 W n 1 R

1

R0 b

c :

(7.16)

(See Exercise 7.5.) Examining (7.16) we can see that e md specializes to e cls when we set W n = 1 0 : Vb An obvious question is which weight matrix W n is best. We will address this question after we derive the asymptotic distribution for a general weight matrix.

7.5

Asymptotic Distribution

We …rst show that the class of minimum distance estimators are consistent for the population parameters when the constraints are valid. Assumption 7.5.1 R0

= c where R is k

q with rank(R) = q:

Theorem 7.5.1 Consistency Under Assumption 1.5.1, Assumption 3.16.1, Assumption 7.5.1, p p and W n ! W > 0; e md ! as n ! 1: Theorem 7.5.1 shows that consistency holds for any weight matrix, so the result includes the CLS estimator. Similarly, the constrained estimators are asymptotically normally distributed.

147

CHAPTER 7. RESTRICTED ESTIMATION Theorem 7.5.2 Asymptotic Normality p Under Assumption 1.5.1, Assumption 6.3.1, Assumption 7.5.1, and W n ! W > 0; p

as n ! 1; where V (W ) = V

W 1

+W

d

n e md

1

! N (0; V (W ))

R R0 W

R R0 W

1

1

1

R

1

R

R0 V

(7.17)

V R R0 W

R0 V R R0 W

1

1

R

1

R

R0 W

1

R0 W

1

1

(7.18)

= Qxx1 Qxx1 :

and V

Theorem 7.5.2 shows that the minimum distance estimator is asymptotically normal for all positive de…nite weight matrices. The asymptotic variance depends on W . The theorem includes the CLS estimator as a special case by setting W = Qxx :

Theorem 7.5.3 Asymptotic Distribution of CLS Estimator Under Assumption 1.5.1, Assumption 6.3.1, and Assumption 7.5.1, as n ! 1 p

where

! N (0; V cls )

Qxx1 R R0 Qxx1 R

V cls = V

+Qxx1 R R0 Qxx1 R

7.6

d

n e cls

1

1

R0 V

V R R0 Qxx1 R

R0 V R R0 Qxx1 R

1

1

R0 Qxx1

R0 Qxx1

E¢cient Minimum Distance Estimator

Theorem 7.5.2 shows that the minimum distance estimators, which include CLS as a special case, are asymptotically normal with an asymptotic covariance matrix which depends on the weight matrix W . The asymptotically optimal weight matrix is the one which minimizes the asymptotic variance V (W ): This turns out to be W = V 1 as is shown in Theorem 7.6.1 below. Since V 1 is unknown this weight matrix cannot be used for a feasible estimator, but we can replace V 1 1 and the asymptotic distribution (and e¢ciency) are unchanged. with a consistent estimate Vb We call the minimum distance estimator setting W n = Vb estimator and takes the form e

emd

=b

Vb R R0 Vb R

1

1

the e¢cient minimum distance

R0 b

c

The asymptotic distribution of (7.19) can be deduced from Theorem 7.5.2.

(7.19)

148

CHAPTER 7. RESTRICTED ESTIMATION Theorem 7.6.1 E¢cient Minimum Distance Estimator Under Assumption 1.5.1, Assumption 6.3.1, and Assumption 7.5.1, p as n ! 1; where V

d

n e emd

! N 0; V

V R R0 V R

=V

1

R0 V :

(7.20)

Since V

(7.21)

V

the estimator (7.19) has lower asymptotic variance than the unrestricted estimator. Furthermore, for any W ; V V (W ) (7.22) so (7.19) is asymptotically e¢cient in the class of minimum distance estimators.

Theorem 7.6.1 shows that the minimum distance estimator with the smallest asymptotic variance is (7.19). One implication is that the constrained least squares estimator is generally ine¢cient. The interesting exception is the case of conditional homoskedasticity, in which case the optimal weight matrix is W = V 0 1 so in this case CLS is an e¢cient minimum distance estimator. Otherwise when the error is conditionally heteroskedastic, there are asymptotic e¢ciency gains by using minimum distance rather than least squares. The fact that CLS is generally ine¢cient is counter-intuitive and requires some re‡ection to understand. Standard intuition suggests to apply the same estimation method (least squares) to the unconstrained and constrained models, and this is the most common empirical practice. But Theorem 7.6.1 shows that this is not the e¢cient estimation method. Instead, the e¢cient minimum distance estimator has a smaller asymptotic variance. Why? The reason is that the least-squares estimator does not make use of the regressor x2i : It ignores the information E (x2i ei ) = 0. This information is relevant when the error is heteroskedastic and the excluded regressors are correlated with the included regressors. Inequality (7.21) shows that the e¢cient minimum distance estimator e has a smaller asymptotic variance than the unrestricted least squares estimator b : This means that estimation is more e¢cient by imposing correct restrictions when we use the minimum distance method.

7.7

Exclusion Restriction Revisited

We return to the example of estimation with a simple exclusion restriction. The model is yi = x01i

1

+ x02i

2

+ ei

with the exclusion restriction 2 = 0: We have introduced three estimators of unconstrained least-squares applied to (7.9), which can be written as

1:

The …rst is

b =Q b 1 Q b 1 11 2 1y 2 :

From Theorem 6.33 and equation (6.20) its asymptotic variance is avar( b 1 ) = Q1112

The second estimator of

11 1

Q12 Q221

21

1 12 Q22 Q21

+ Q12 Q221

1 22 Q22 Q21

is the CLS estimator, which can be written as e

1;cls

b 1Q b 1y : =Q 11

Q1112 :

149

CHAPTER 7. RESTRICTED ESTIMATION

Its asymptotic variance can be deduced from Theorem 7.5.3, but it is simpler to apply the CLT directly to show that avar( e 1;cls ) = Q111 11 Q111 : (7.23) The third estimator of

1

is the e¢cient minimum distance estimator. Applying (7.19), it equals e

where we have partitioned

1;md

Vb

1 Vb 12 Vb 22 b 2

= b1

=

"

From Theorem 7.6.1 its asymptotic variance is

Vb 11 Vb 12 Vb 21 Vb 22

avar( e 1;md ) = V 11

#

(7.24)

:

V 12 V 221 V 21 :

(7.25)

In general, the three estimators are di¤erent, and they have di¤erent asymptotic variances. It is quite instructive to compare the asymptotic variances of the CLS and unconstrained leastsquares estimators to assess whether or not the constrained estimator is necessarily more e¢cient than the unconstrained estimator. First, consider the case of conditional homoskedasticity. In this case the two covariance matrices simplify to avar( b 1 ) = 2 Q1112 and

avar( e 1;cls ) =

2

Q111 :

If Q12 = 0 (so x1i and x2i are orthogonal) then these two variance matrices equal and the two estimators have equal asymptotic e¢ciency. Otherwise, since Q12 Q221 Q21 0; then Q11 Q11 Q12 Q221 Q21 ; and consequently Q111

2

Q11

Q12 Q221 Q21

1

2

:

This means that under conditional homoskedasticity, e 1;cls has a lower asymptotic variance matrix than b 1 : Therefore in this context, constrained least-squares is more e¢cient than unconstrained least-squares. This is consistent with our intuition that imposing a correct restriction (excluding an irrelevant regressor) improves estimation e¢ciency. However, in the general case of conditional heteroskedasticity this ranking is not guaranteed. In fact what is really amazing is that the variance ranking can be reversed. The CLS estimator can have a larger asymptotic variance than the unconstrained least squares estimator. To see this let’s use the simple heteroskedastic example from Section 6.4. In that example, 7 3 1 Q11 = Q22 = 1; Q12 = ; 11 = 22 = 1; and 12 = : We can calculate that Q11 2 = and 2 8 4 2 3 avar( e 1;cls ) = 1 avar( b 1 ) =

(7.26) (7.27)

5 (7.28) avar( e 1;md ) = : 8 Thus the restricted least-squares estimator e 1;cls has a larger variance than the unrestricted leastsquares estimator b 1 ! The minimum distance estimator has the smallest variance of the three, as expected.

150

CHAPTER 7. RESTRICTED ESTIMATION

What we have found is that when the estimation method is least-squares, deleting the irrelevant variable x2i can actually increase estimation variance; or equivalently, adding an irrelevant variable can actually decrease the estimation variance. To repeat this unexpected …nding, we have shown in a very simple example that it is possible for least-squares applied to the short regression (7.10) to be less e¢cient for estimation of 1 than least-squares applied to the long regression (7.9), even though the constraint 2 = 0 is valid! This result is strongly counter-intuitive. It seems to contradict our initial motivation for pursuing constrained estimation – to improve estimation e¢ciency. It turns out that a more re…ned answer is appropriate. Constrained estimation is desirable, but not constrained least-squares estimation. While least-squares is asymptotically e¢cient for estimation of the unconstrained projection model, it is not an e¢cient estimator of the constrained projection model.

7.8

Variance and Standard Error Estimation

The asymptotic covariance matrix (7.20) may be estimated by replacing V estimates such as Vb . This variance estimator is then Vb

= Vb

Vb R R0 Vb R

1

with a consistent

R0 Vb :

(7.29)

:

(7.30)

We can calculate standard errors for any linear combination h0 e so long as h does not lie in the range space of R. A standard error for h0 e is

7.9

Misspeci…cation

s(h0 e ) = n

h Vb h

1 0

1=2

What are the consequences for the constrained estimator e if the constraint (7.1) is incorrect? To be speci…c, suppose that R0 = c where c is not necessarily equal to c: This situation is a generalization of the analysis of “omitted variable bias” from Section 3.21, where we found that the short regression (e.g. (7.11)) is estimating a di¤erent projection coe¢cient than the long regression (e.g. (7.9)). One mechanical answer is that we can use the formula (7.16) for the minimum distance estimator to …nd that p 1 e W 1 R R0 W 1 R (c c) : md ! md = 1

(c c), shows that imposing an incorrect constraint leads The second term, W 1 R R0 W 1 R to inconsistency – an asymptotic bias. We can call the limiting value md the minimum-distance projection coe¢cient or the pseudo-true value implied by the restriction. However, we can say more. For example, we can describe some characteristics of the approximating projections. The CLS estimator projection coe¢cient has the representation cls

= argmin E yi R

0

=c

x0i

2

;

the best linear predictor subject to the constraint (7.1). The minimum distance estimator converges to 0 0) W ( 0) md = argmin ( R0 =c

151

CHAPTER 7. RESTRICTED ESTIMATION

where 0 is the true coe¢cient. That is, md is the coe¢cient vector satisfying (7.1) closest to the true value ¬n the weighted Euclidean norm. These calculations show that the constrained estimators are still reasonable in the sense that they produce good approximations to the true coe¢cient, conditional on being required to satisfy the constraint. We can also show that e md has an asymptotic normal distribution. The trick is to de…ne the pseudo-true value 1 (c c) : W n 1 R R0 W n 1 R n = Then p

n e md

n

=

p

=

I

1p W n 1 R R0 W n 1 R n R0 b 1 0 p R W n 1 R R0 W n 1 R n b

n b

d

! I

W

1

R R0 W

1

1

R

c

R0 N (0; V )

= N (0; V (W )) : In particular

p

(7.31) d

n e emd

! N 0; V

n

:

This means that even when the constraint (7.1) is misspeci…ed, the conventional covariance matrix estimator (7.29) and standard errors (7.30) are appropriate measures of the sampling variance, though the distributions are centered at the pseudo-true values (or projections) n rather than : The fact that the estimators are biased is an unavoidable consequence of misspeci…cation. There is another way of representing an asymptotic distribution for the estimator under misspeci…cation based on the concept of local alternatives. It is a technical device which might seem a bit arti…cial, but it is a powerful technique which yields useful distributional approximations in a wide variety of contexts. The idea is to index the true coe¢cient n by n; and suppose that R0

n

=c+ n

1=2

:

(7.32)

The asymptotic theory is then derived as n ! 1 under the sequence of probability distributions with the coe¢cients n . The expression (7.32) speci…es that n does not satisfy (7.1), but the deviation from the constraint is n 1=2 which depends on and the sample size. The choice to make the deviation of this form is precisely so that the localizing parameter appears in the asymptotic distribution but does not dominate it. To see this, …rst observe that since n is the true coe¢cient value, then yi = x0i n + ei and p

n

n b

n

=

1X xi x0i n i=1

!

1

n

1 X p xi ei n i=1

!

d

! N (0; V )

as conventional. Then under (7.32), e so p

md

n e md

= b

= b n

=

W n 1 R R0 W n 1 R

1

W n 1 R R0 W n 1 R

1

I

R0 b

R0 b

W n 1 R R0 W n 1 R

d

! N (0; V ) + W

= N ( ; V (W ))

c

1

1

+ W n 1 R R0 W n 1 R

R0

R R0 W

p 1

n b

R

n

1

n

1=2

+ W n 1 R R0 W n 1 R

1

1

(7.33)

152

CHAPTER 7. RESTRICTED ESTIMATION where =W

1

R R0 W

1

1

R

:

The asymptotic distribution (7.33) is an alternative way of expressing the sampling distribution of the restricted estimator under misspeci…cation. The distribution (7.33) contains an asymptotic bias component : The approximation is not fundamentally di¤erent from (7.31), they both have the same asymptotic variances, and both re‡ect the bias due to misspeci…cation. The di¤erence is that (7.31) puts the bias on the left-side of the convergence arrow, while (7.33) has the bias on the right-side. There is no substantive di¤erence between the two, but (7.33) is more convenient for some purposes, such as the analysis of the power of tests, as we will explore in the next chapter.

7.10

Nonlinear Constraints

In some cases it is desirable to impose nonlinear constraints on the parameter vector can be written as r( ) = 0 Rk

. They (7.34)

Rq :

where r : ! This includes the linear constraints (7.1) as a special case. An example of (7.34) which cannot be written as (7.1) is 1 2 = 1; which is (7.34) with r( ) = 1 2 1: The minimum distance estimator of subject to (7.34) solves the minimization problem e = argmin Jn ( )

(7.35)

r( )=0

where

0

Jn ( ) = n b

The solution minimizes the Lagrangian

Wn b

1 L( ; ) = Jn ( ) + 2

0

:

r( )

(7.36)

over ( ; ): Computationally, there is no explicit expression for the solution e so it must be found numerically. Algorithms to numerically solve (7.35) are known as constrained optimization methods, and are available in programming languages including Matlab, Gauss and R.

Assumption 7.10.1 r( ) = 0 with rank(R) = q; where R =

@ r( )0 : @

The asymptotic distribution is a simple generalization of the case of a linear constraint, but the proof is more delicate. Theorem 7.10.1 Under Assumption 1.5.1, Assumption 6.3.1, Assumption p 7.10.1, and W n ! W > 0; for e de…ned in (7.35) , p

n e

d

! N (0; V (W ))

as n ! 1; where V (W ) is de…ned in (7.18). V (W ) is minimized with W = V 1 ; in which case the asymptotic variance is V

=V

V R R0 V R

1

R0 V :

153

CHAPTER 7. RESTRICTED ESTIMATION

The asymptotic variance matrix for the e¢cient minimum distance estimator can be estimated by

where

Vb

1

b 0 Vb R b b R Vb R

= Vb

b = @ r( e )0 : R @

b 0 Vb R

Standard errors for the elements of e are the square roots of the diagonal elements of n

7.11

Inequality Constraints

Inequality constraints on the parameter vector

1V b

:

take the form

r( )

0

for some function r : Rk ! Rq : The most common example is a non-negative constraint 1

0:

The constrained least-squares and minimum distance estimators can be written as e

and

cls

e

= argmin SSEn ( )

md

(7.37)

r( ) 0

= argmin Jn ( ) :

(7.38)

r( ) 0

Except in special cases the constrained estimators do not have simple algebraic solutions. An important exception is when there is a single non-negativity constraint, e.g. 1 0 with q = 1: In this case the constrained estimator can be found by two-step approach. First compute the uncontrained estimator b . If b 1 0 then e = b : Second, if b 1 < 0 then impose 1 = 0 (eliminate the regressor X1 ) and re-estimate. This yields the constrained least-squares estimator. While this method works when there is a single non-negativity constraint, it does not immediately generalize to other contexts. The computational problems (7.37) and (7.38) are examples of quadratic programming problems. Quick and easy computer algorithms are available in programming languages including Matlab, Gauss and R. Inference on inequality-constrained estimators is unfortunately quite challenging. The conventional asymptotic theory gives rise to the following dichotomy. If the true parameter satis…es the strict inequality r( ) > 0, then asymptotically the estimator is not subject to the constraint and the inequality-constrained estimator has an asymptotic distribution equal to the unconstrained case. However if the true parameter is on the boundary, e.g. r( ) = 0, then the estimator has a truncated structure. This is easiest to see in the one-dimensional case. If we have an estimator ^ which p d satis…es n ^ ! Z = N (0; V ) and = 0; then the constrained estimator ~ = max[ ^ ; 0] p d will have the asymptotic distribution n ~ ! max[Z; 0]; a “half-normal” distribution.

7.12

Constrained MLE

Recall that the log-likelihood function (4.39) for the normal regression model is log L( ;

2

)=

n log 2 2

1

2

2

2

SSEn ( ):

154

CHAPTER 7. RESTRICTED ESTIMATION

The constrained maximum likelihood estimator (CMLE) ( b cmle ; ^ 2cmle ) maximizes log L( ; 2 ) subject to the constraint (7.34) Since log L( ; 2 ) is a function of only through the sum of squared errors SSEn ( ); maximizing the likelihood is identical to minimizing SSEn ( ). Hence b cmle = b cls and ^ 2cmle = ^ 2cls .

7.13

Technical Proofs*

Proof of Theorem 7.6.1, Equation (7.22). Let R? be a full rank k (k q) matrix satisfying R0? V R = 0 and then set C = [R; R? ] which is full rank and invertible. Then we can calculate that C 0V C = =

R0 V R R0 V R? R0? V R R0? V R? 0 0 0 R0? V R?

and R0 V (W )R R0 V (W )R? R0? V (W )R R0? V (W )R?

C 0 V (W )C =

0 0 1 0 R0? V R? + R0? W R (R0 W R) R0 V R (R0 W R)

=

1

R0 W R?

:

Thus C 0 V (W )

V

C = C 0 V (W )C

C 0V C

0 0 R0? W R (R0 W R)

=

Since C is invertible it follows that V (W )

V

1

0 R V R (R0 W R) 0

1

R0 W R?

0

0 which is (7.22).

Proof of Theorem 7.10.1. For simplicity, we assume that the constrained estimator is consistent e p! . This can be shown with more e¤ort, but requires a deeper treatment than appropriate for this textbook. For each element rj ( ) of the q-vector r( ); by the mean value theorem there exists a j on the line segment joining e and such that Let Rn be the k

q matrix

rj ( e ) = rj ( ) + @ r1 ( @

Rn = p Since e !

it follows that

1)

@ rj ( @

@ r2 ( @

0 j)

:

@ rq ( @

2)

(7.39)

q)

:

p

p j

e

! , and by the CMT, Rn ! R: Stacking the (7.39), we obtain r( e ) = r( ) + Rn0 e

:

Since r( e ) = 0 by construction and r( ) = 0 by Assumption 7.5.1, this implies 0 = Rn0 e

:

(7.40)

155

CHAPTER 7. RESTRICTED ESTIMATION The …rst-order condition for (7.36) is Wn b

e =R e e:

Premultiplying by R 0 W n 1 ; inverting, and using (7.40), we …nd

Thus

e = R 0W 1R e n n e

=

1

Rn0 b

f e R 0W 1H W n 1R n n

I

1

e = R 0W 1R e n n 1

b

Rn0

From Theorem 6.3.2 and Theorem 6.7.1 we …nd p

n e

=

I d

e R 0W 1R e W n 1R n n

! I

W

1

R R0 W

= N (0; V (W )) :

1

1

R

Rn0 b

Rn0 1

:

:

p

n b

R0 N (0; V )

156

CHAPTER 7. RESTRICTED ESTIMATION

Exercises Exercise 7.1 In the model y = X 1 1 + X 2 2 + e; show directly from de…nition (7.3) that the CLS estimate of = ( 1 ; 2 ) subject to the constraint that 2 = 0 is the OLS regression of y on X 1: Exercise 7.2 In the model y = X 1 1 + X 2 2 + e; show directly from de…nition (7.3) that the CLS estimate of = ( 1 ; 2 ); subject to the constraint that 1 = c (where c is some given vector) is the OLS regression of y X 1 c on X 2 : Exercise 7.3 In the model y = X 1 1 + X 2 2 + e; with X 1 and X 2 each n estimate of = ( 1 ; 2 ); subject to the constraint that 1 = 2:

k; …nd the CLS

Exercise 7.4 Verify that for e de…ned in (7.8) that R0 e = c: Exercise 7.5 Verify (7.16).

b xx equals the CLS Exercise 7.6 Verify that the minimum distance estimator e with W n = Q estimator. Exercise 7.7 Prove Theorem 7.5.1. Exercise 7.8 Prove Theorem 7.5.2. Exercise 7.9 Prove Theorem 7.5.3. (Hint: Use that CLS is a special case of Theorem 7.5.2.) Exercise 7.10 Verify that (7.20) is V (W ) with W = V Exercise 7.11 Prove (7.21). Hint: Use (7.20). Exercise 7.12 Verify (7.23), (7.24) and (7.25) Exercise 7.13 Verify (7.26), (7.27), and (7.28).

1

:

Chapter 8

Hypothesis Testing 8.1

Hypotheses and Tests

It is often the goal of an empirical investigation to determine if one variable a¤ects another. Returning to our example of wage determination, we might be interested if union membership a¤ects wages. Equivalently, we might ask if the hypothesis “Union membership does not a¤ect wages” is true or false. In hypothesis testing, we are interest in testing if a speci…c restriction on the parameters is compatible with the observed data. Letting be the coe¢cient in a wage regression for union membership, the hypothesis that union membership has no e¤ect on mean wages is the restriction = 0: Hypothesis testing is about making a decision if the restriction = 0 is true or false. In general, a hypothesis is a statement (or assertion) that a restriction is true, where a restriction takes the form 2 0 with 0 is a strict subset of a pararameter space . De…nition 1 A hypothesis is a statement that

For every hypothesis 2 is the complement of 0 in

2

0

.

the statement 2 c0 is also a hypothesis, where c0 = f 2 . We give a hypothesis and its complement special names. 0

De…nition 2 The null hypothesis is H0 : hypothesis is its complement H1 : 2 c0 :

2

0

:

2 =

and the alternative

In the example given previously, the alternative hypothesis is that union membership has an e¤ect on mean wages, or 6= 0: A hypothesis can either be true or not true. The goal of hypothesis testing is to provide evidence concerning the truth of the null hypothesis versus its alternative. A hypothesis test makes one of two decisions based on the data: either “Accept H0 ” or “Reject H0 ”. Take again the example about union membership and examine the wage regression reported in Table 5.1. We see that the coe¢cient for “Male Union Member” is 0.095 (a wage premium of 9.5%) with a standard error of 0.020. Given the magnitude of this estimate, it seems unlikely that the true coe¢cient could be zero, and so we may be inclined to reject the hypothesis that union membership does not a¤ect wages for males. However, we can also see that the coe¢cient for “Female Union Member” is 0.022 with a standard error of 0.020. While the point estimate suggests a wage premium of 2.2%, the standard error suggests that it is also plausible that the true 157

0g

158

CHAPTER 8. HYPOTHESIS TESTING

coe¢cient could be zero and the point estimate is merely sampling error. In this case what decision should we make? A hypothesis test consists of a real-valued test statistic Tn = Tn ((y1 ; x1 ) ; :::; (yn ; xn )) ; a critical value c, plus the decision rule 1. Accept H0 if Tn < c; 2. Reject H0 if Tn

c:

The test statistic Tn should be designed so that small values of Tn are likely when H0 is true and large values of Tn are likely when H1 is true. For example, for a test of H0 : = 0 against H1 : 6= 0 the standard test statistic is the absolute t-statistic Tn = jtn ( 0 )j ; as we expect Tn to have a well-behaved distribution = 0 ; and to be large when 6= 0 : Given the two possible states of the world (H0 or H1 ) and the two possible decisions (Accept H0 or Reject H0 ), there are four possible pairings of states and decisions as is depicted in the following chart. Hypthesis Testing Decisions H0 true H1 true

Accept H0 Correct Decision Type II Error

Reject H0 Type I Error Correct Decision

Hypothesis tests are useful if they avoid making errors. There are two possible errors: (1) Rejecting H0 when H0 is true; and (2) Accepting H0 when H0 is false. These two errors are called Type I and Type II errors, respectively. As the events are random it is constructive to evaluate the tests based on the probability of their making an error. For a given test we de…ne the rejection probability function as the probability of rejecting the null hypothesis n(

) = Pr (Reject H0 j ) c j ):

= Pr (Tn

The rejection probability in general depends on the unknown parameter and the sample size n: For parameter values in the null hypothesis, 2 0 ; n ( ) is the probability of making a Type I error. We also call n ( ) the power function of the test, as for parameter values in the alternative, 2 c0 , n ( ) equals 1 minus the probability of a Type II error. For the reasons discussed in Chapter 6, in typical econometric models the exact sampling distribution of statistics such as Tn is unknown and hence n ( ) is unknown. Therefore we typically rely on asymptotic approximations which allow a more precise characterization. In particular, we focus on the asymptotic rejection probability ( ) = lim

n(

n!1

):

Therefore, it is convenient to select a test statistic Tn which has a well speci…ed asymptotic disd tribution under H0 , that is, Tn ! under H0 : Furthermore, if the distribution F of does not depend on 2 0 we say that Tn is asymptotically pivotal. In this case, ( ) =

lim Pr (Tn

n!1

= Pr ( = 1

cj )

c) F (c)

is only a function of c. For example, if Tn is the absolute t-statistic, then by Theorem 6.11.1, under d H0 ; Tn ! jZj where Z N(0; 1) and thus F (c) = (c); the symmetrized normal distribution function de…ned in (6.40).

159

CHAPTER 8. HYPOTHESIS TESTING

Given a test statistic Tn , how should we select the critical value c? Larger values of c mean that the test rejects less frequently, which decreases the probability of a Type I error but increases the probability of a Type II error. How can we balance one against the other? The dominant approach is to give special priority to the null hypothesis, and select c so that the Type I error probability is controlled at a speci…ed level, meaning that we select a signi…cance level 2 (0; 1) and then pick c so that ( ) for all 2 0 : When the test statistic Tn is asymptotically pivotal with distribution F this is accomplished by selecting c so that 1 F (c) = : There is no objective scienti…c basis for choice of signi…cance level : However, the common practice is to set = :05 (5%). The intention is that Type I errors should be relatively unlikely – that the decision “Reject H0 ” has scienti…c strength. The decision “Reject H0 ” means that the evidence is inconsistent with the null hypothesis, in the sense that it is relatively unlikely (1 in 20) that data generated by the null hypothesis would yield the observed test result. In contrast, the decision “Accept H0 ” is not a strong statement. It does not mean that the evidence supports H0 , only that there is insu¢cient evidence to reject H0 . Because of this, it is probably more accurate to use the label “Do not Reject H0 ” instead of “Accept H0 ”. When a test rejects H0 at the 5% signi…cance level it is common to say that the statistic is statistically signi…cant and if the test accepts H0 it is common to say that the statistic is statistically insigni…cant. It is helpful to remember that this is simply a way of saying “Using the statistic Tn , the hypothesis H0 can [cannot] be rejected at the asymptotic 5% level.” When the null hypothesis H0 : = 0 is rejected it is common to say that the coe¢cient is statistically signi…cant, because the test has shown that the coe¢cient is not equal to zero. Let us return to the example about the union wage premium. The absolute t-statistic for the coe¢cient on “Male Union Member” is 0:095=0:020 = 4:75; which is greater than the 5% asymptotic critical value of 1.96. Therefore we can reject the hypothesis that union membership does not a¤ect wages for men. However, the absolute t-statistic for the coe¢cient on “Female Union Member” is 0:022=0:020 = 1:10; which is less than 1.96 and therefore we do not reject the hypothesis that union membership does not a¤ect wages for women. We thus say that the e¤ect of union membership for men is statistically signi…cant, while membership for women is not statistically signi…cant. Consider another question: Does marriage status a¤ect wages? To test the hypothesis that marriage status has no e¤ect on wages, we examine the t-statistics for the coe¢cients on “Married Male” and “Married Female” in Table 5.1, which are 0:180=0:008 = 22:5 and 0:016=0:008 = 2:0; respectively. Both exceed the asymptotic 5% critical value of 1.96, so we reject the hypothesis for both men and women. But the statistic for men is exceptionally high, and that for women is only slightly above the critical value. Suppose in contrast that the t-statistic had been 1.9, which is less than the critical value, leading to the decision “Accept H0 ” rather than “Reject H0 ”. Should we really be making a di¤erent decision if the t-statistic is 1.9 rather than 2.0? The di¤erence in values is small, shouldn’t the di¤erence in the decision be also small? Thinking through these examples it seems unsatisfactory to simply report “Accept H0 ” or “Reject H0 ”. These two decisions do not summarize the evidence. Instead, the magnitude of the statistic Tn suggests a “degree of evidence” against H0 : How can we take this into account? The answer is to report what is known as the asymptotic p-value pn = 1

F (Tn ):

Since the distribution function F is monotonically increasing, the p-value is a monotonically decreasing function of Tn and is an equivalent test statistic. Instead of rejecting H0 at the signi…cance level if Tn c ; we can reject H0 if pn : Thus it is su¢cient to report pn ; and let the reader decide. Furthermore, the asymptotic p-value has a very convenient asymptotic null distribution. Since

160

CHAPTER 8. HYPOTHESIS TESTING d

Tn !

d

under H0 ; then pn = 1 Pr (1

F (Tn ) ! 1 F( )

F ( ), which has the distribution

u) = Pr (1 = 1

u

F ( ))

Pr

= 1

F F

= 1

(1

F 1

(1

1

(1

u)

u)

u)

= u; d

which is the uniform distribution on [0; 1]: Thus pn ! U[0; 1]: This means that the “unusualness” of pn is easier to interpret than the “unusualness” of Tn : The implication is that the best empirical practice is to compute and report the asymptotic pvalue pn rather than simply the test statistic Tn or the binary decision Accept/Reject. The p-value is a simple statistic, easy to interpret, and contains more information than the other choices. For example, consider the tests for the e¤ect of marriage status on wages. The p-value for men is 0.000 which we can intrepret as a very strong rejection of the hypothesis of no e¤ect, while the p-value for women is 0.046, which we can describe as “borderline signi…cant”. We now summarize the main features of hypothesis testing. 1. Select a signi…cance level : d

2. Select a test statistic Tn with asymptotic distribution Tn ! 3. Set the critical value c so that 1

under H0 :

F (c) = ; where F is the distribution function of :

4. Calculate the asymptotic p-value pn = 1

F (Tn ):

5. Reject H0 if Tn

:

c; or equivalently pn

6. Accept H0 if Tn < c; or equivalently pn > : 7. Report pn to summarize the evidence concerning H0 versus H1 :

8.2

t tests

The most commonly applied statistical tests are hypotheses on individual coe¢cients and can be written in the form H0 : = 0

= h( ), (8.1)

where 0 is some pre-speci…ed value. Quite typically, 0 = 0; as interest focuses on whether or not a coe¢cient equals zero, but this is not the only possibility. For example, interest may focus on whether an elasticity equals 1, in which case we may wish to test H0 : = 1: As we described in the previous section, tests of H0 typically are based on the t-statistic tn ( ) =

b

s(b)

where b is the point estimate and s(b) is its standard error. The test will depend on whether the alternative hypothesis is one-sided or two-sided. The two-sided alternative is H1 :

6=

0

(8.2)

which is appropriate for general tests of signi…cance. In this case the test statistic is the absolute value of the t-statistic b 0 tn = jtn ( 0 )j = : s(b)

161

CHAPTER 8. HYPOTHESIS TESTING Since Theorem 6.11.1 established that when

=

0;

d

tn ! jZj where Z N(0; 1), then asymptotic critical values can be taken from the normal distribution table. In particular, the asymptotic 5% critical value is c = 1:96: Also, the asymptotic p-value for H0 against H1 is pn = 2 (1

(tn )) :

Theorem 8.2.1 Under Assumption 1.5.1, Assumption 6.3.1, Assumption 6.9.1, and H0 ; d jtn ( 0 )j ! jZj and for c satisfying

= 2 (1

(c)) ; c j H0 ) !

Pr (jtn ( 0 )j so the test “Reject H0 if jtn ( 0 )j

c” has asymptotic signi…cance level

:

We described (8.2) as a “two-sided alternative”. Sometimes we are interested in testing for one-sided alternatives such as H1 : > 0 (8.3) or H1 :


0: The test for H0 is to reject if Tn = ^=s(^) > c where c is picked so that Type I error is : You do this as follows. Using the nonparametric bootstrap, you generate bootstrap samples, calculate the estimates ^ on these samples and then calculate Tn = ^ =s(^ ): Let qn (:95) denote the 95% quantile of Tn . You replace c with qn (:95); and thus reject H0 if Tn = ^=s(^) > qn (:95): What is wrong with this procedure? Exercise 10.6 Suppose that in an application, ^ = 1:2 and s(^) = :2: Using the non-parametric bootstrap, 1000 samples are generated from the bootstrap distribution, and ^ is calculated on each sample. The ^ are sorted, and the 2.5% and 97.5% quantiles of the ^ are .75 and 1.3, respectively. (a) Report the 95% Efron Percentile interval for : (b) Report the 95% Alternative Percentile interval for : (c) With the given information, can you report the 95% Percentile-t interval for ? Exercise 10.7 The data…le hprice1.dat contains data on house prices (sales), with variables listed in the …le hprice1.pdf. Estimate a linear regression of price on the number of bedrooms, lot size, size of house, and the colonial dummy. Calculate 95% con…dence intervals for the regression coe¢cients using both the asymptotic normal approximation and the percentile-t bootstrap.

Chapter 11

NonParametric Regression 11.1

Introduction

When components of x are continuously distributed then the conditional expectation function E (yi j xi = x) = m(x) can take any nonlinear shape. Unless an economic model restricts the form of m(x) to a parametric function, the CEF is inherently nonparametric, meaning that the function m(x) is an element of an in…nite dimensional class. In this situation, how can we estimate m(x)? What is a suitable method, if we acknowledge that m(x) is nonparametric? There are two main classes of nonparametric regression estimators: kernel estimators, and series estimators. In this chapter we introduce kernel methods. To get started, suppose that there is a single real-valued regressor xi : We consider the case of vector-valued regressors later.

11.2

Binned Estimator

For clarity, …x the point x and consider estimation of the single point m(x). This is the mean of yi for random pairs (yi ; xi ) such that xi = x: If the distribution of xi were discrete then we could estimate m(x) by taking the average of the sub-sample of observations yi for which xi = x: But when xi is continuous then the probability is zero that xi exactly equals any speci…c x. So there is no sub-sample of observations with xi = x and we cannot simply take the average of the corresponding yi values. However, if the CEF m(x) is continuous, then it should be possible to get a good approximation by taking the average of the observations for which xi is close to x; perhaps for the observations for which jxi xj h for some small h > 0: Later we will call h a bandwidth. This estimator can be written as Pn 1 (jxi xj h) yi (11.1) m(x) b = Pi=1 n xj h) i=1 1 (jxi where 1( ) is the indicator function. Alternatively, (11.1) can be written as m(x) b =

where

Notice that

Pn

i=1 wi (x)

n X

wi (x)yi

i=1

1 (jxi xj h) wi (x) = Pn : xj h) j=1 1 (jxj

= 1; so (11.2) is a weighted average of the yi . 200

(11.2)

CHAPTER 11. NONPARAMETRIC REGRESSION

201

Figure 11.1: Scatter of (yi ; xi ) and Nadaraya-Watson regression To visualize, Figure 11.1 displays a scatter plot of 100 observations on a random pair (yi ; xi ) generated by simulation1 . (The observations are displayed as the open circles.) The estimator (11.1) of the CEF m(x) at x = 2 with h = 1=2 is the average of the yi for the observations such that xi falls in the interval [1:5 xi 2:5]: (Our choice of h = 1=2 is somewhat arbitrary. Selection of h will be discussed later.) The estimate is m(2) b = 5:16 and is shown on Figure 11.1 by the …rst solid square. We repeat the calculation (11.1) for x = 3; 4, 5, and 6, which is equivalent to partitioning the support of xi into the regions [1:5; 2:5]; [2:5; 3:5]; [3:5; 4:5]; [4:5; 5:5]; and [5:5; 6:5]: These partitions are shown in Figure 11.1 by the verticle dotted lines, and the estimates (11.1) by the solid squares. These estimates m(x) b can be viewed as estimates of the CEF m(x): Sometimes called a binned estimator, this is a step-function approximation to m(x) and is displayed in Figure 11.1 by the horizontal lines passing through the solid squares. This estimate roughly tracks the central tendency of the scatter of the observations (yi ; xi ): However, the huge jumps in the estimated step function at the edges of the partitions are disconcerting, counter-intuitive, and clearly an artifact of the discrete binning. If we take another look at the estimation formula (11.1) there is no reason why we need to evaluate (11.1) only on a course grid. We can evaluate m(x) b for any set of values of x: In particular, we can evaluate (11.1) on a …ne grid of values of x and thereby obtain a smoother estimate of the CEF. This estimator with h = 1=2 is displayed in Figure 11.1 with the solid line. This is a generalization of the binned estimator and by construction passes through the solid squares. The bandwidth h determines the degree of smoothing. Larger values of h increase the width of the bins in Figure 11.1, thereby increasing the smoothness of the estimate m(x) b as a function of x. Smaller values of h decrease the width of the bins, resulting in less smooth conditional mean estimates.

11.3

Kernel Regression

One de…ciency with the estimator (11.1) is that it is a step function in x, as it is discontinuous at each observation x = xi : That is why its plot in Figure 11.1 is jagged. The source of the dis1

The distribution is xi

N (4; 1) and yi j xi

N (m(xi ); 16) with m(x) = 10 log(x):

202

CHAPTER 11. NONPARAMETRIC REGRESSION

continuity is that the weights wi (x) are constructed from indicator functions, which are themselves discontinuous. If instead the weights are constructed from continuous functions then the CEF estimator will also be continuous in x: To generalize (11.1) it is useful to write the weights 1 (jxi xj h) in terms of the uniform density function on [ 1; 1] 1 k0 (u) = 1 (juj 1) : 2 Then xi x xi x 1 (jxi xj h) = 1 1 = 2k0 : h h and (11.1) can be written as Pn

xi

i=1 k0

m(x) b =

Pn

i=1 k0

x

h xi x h

yi :

(11.3)

The uniform density k0 (u) is a special case of what is known as a kernel function.

De…nition 11.3.1 A second-order kernel function satis…es 0 R1 R 1 k(u) 2 2 k(u) < 1; k(u) = k( u); 1 k(u)du = 1 and k = 1 u k(u)du < 1: Essentially, a kernel function is a probability density function which is bounded and symmetric about zero. A generalization of (11.1) is obtained by replacing the uniform kernel with any other kernel function: Pn xi x yi i=1 k h : (11.4) m(x) b = Pn xi x i=1 k h The estimator (11.4) also takes the form (11.2) with

x h wi (x) = : Pn xj x j=1 k h k

xi

The estimator (11.4) is known as the Nadaraya-Watson estimator, the kernel regression estimator, or the local constant estimator. The bandwidth h plays the same role in (11.4) as it does in (11.1). Namely, larger values of h will result in estimates m(x) b which are smoother in x; and smaller values of h will result in estimates which are more erratic. It might be helpful to consider the two extreme cases h ! 0 and h ! 1: As h ! 0 we can see that m(x b i ) ! yi (if the values of xi are unique), so that m(x) b is simply the scatter of yi on xi : In contrast, as h ! 1 then for all x; m(x) b ! y; the sample mean, so that the nonparametric CEF estimate is a constant function. For intermediate values of h; m(x) b will lie between these two extreme cases. The uniform density is not a good kernel choice as it produces discontinuous CEF estimates: To obtain a continuous CEF estimate m(x) b it is necessary for the kernel k(u) to be continuous. The two most commonly used choices are the Epanechnikov kernel k1 (u) =

3 1 4

u2 1 (juj

1)

203

CHAPTER 11. NONPARAMETRIC REGRESSION and the normal or Gaussian kernel u2 2

1 k (u) = p exp 2

:

For computation of the CEF estimate (11.4) the scale of the kernel is not important so long as u the bandwidth is selected appropriately. That is, for any b > 0; kb (u) = b 1 k is a valid kernel b function with the identical shape as k(u): Kernel regression with the kernel k(u) and bandwidth h is identical to kernel regression with the kernel kb (u) and bandwidth h=b: The estimate (11.4) using the Epanechnikov kernel and h = 1=2 is also displayed in Figure 11.1 with the dashed line. As you can see, this estimator appears to be much smoother than that using the uniform kernel. Two important constants associated with a kernel function k(u) are its variance 2k and roughness Rk , which are de…ned as Z 1 2 u2 k(u)du (11.5) k = 1 Z 1 Rk = k(u)2 du (11.6) 1

Some common kernels and their roughness and variance values are reported in Table 9.1. Table 9.1: Common Second-Order Kernels

11.4

Kernel Uniform Epanechnikov Biweight Triweight

Equation k0 (u) = 12 1 (juj 1) k1 (u) = 43 1 u2 1 (juj 1) 2 15 1 u2 1 (juj 1) k2 (u) = 16 3 35 k3 (u) = 32 1 u2 1 (juj 1)

Gaussian

k (u) =

p1 2

u2 2

exp

Rk 1=2 3=5 5=7 350=429 p 1=2

2 k

1=3 1=5 1=7 1=9 1

Local Linear Estimator

The Nadaraya-Watson (NW) estimator is often called a local constant estimator as it locally (about x) approximates the CEF m(x) as a constant function. One way to see this is to observe that m(x) b solves the minimization problem m(x) b = argmin

n X

xi

k

x

h

i=1

(yi

)2 :

This is a weighted regression of yi on an intercept only. Without the weights, this estimation problem reduces to the sample mean. The NW estimator generalizes this to a local mean. This interpretation suggests that we can construct alternative nonparametric estimators of the CEF by alternative local approximations. Many such local approximations are possible. A popular choice is the local linear (LL) approximation. Instead of approximating m(x) locally as a constant, the local linear approximation approximates the CEF locally by a linear function, and estimates this local approximation by locally weighted least squares. Speci…cally, for each x we solve the following minimization problem n

n o X b (x); b (x) = argmin k ;

i=1

xi

x h

(yi

(xi

x))2 :

204

CHAPTER 11. NONPARAMETRIC REGRESSION

Figure 11.2: Scatter of (yi ; xi ) and Local Linear …tted regression The local linear estimator of m(x) is the estimated intercept m(x) b = b (x)

and the local linear estimator of the regression derivative rm(x) is the estimated slope coe¢cient d rm(x) = b (x):

Computationally, for each x set

1

z i (x) = and ki (x) = k

xi

x

xi

x

:

h

Then b (x) b (x)

= =

n X i=1 0

0

ki (x)z i (x)z i (x)

Z KZ

1

Z 0 Ky

!

1 n X

ki (x)z i (x)yi

i=1

where K = diagfk1 (x); :::; kn (x)g: To visualize, Figure 11.2 displays the scatter plot of the same 100 observations from Figure 11.1, divided into three regions depending on the regressor xi : [1; 3]; [3; 5]; [5; 7]: A linear regression is …t to the observations in each region, with the observations weighted by the Epanechnikov kernel with h = 1: The three …tted regression lines are displayed by the three straight solid lines. The values of these regression lines at x = 2; x = 4 and x = 6; respectively, are the local linear estimates m(x) b at x = 2; 4, and 6. This estimation is repeated for all x in the support of the regressors, and plotted as the continuous solid line in Figure 11.2. One interesting feature is that as h ! 1; the LL estimator approaches the full-sample linear least-squares estimator m(x) b ! ^ + ^ x. That is because as h ! 1 all observations receive equal

205

CHAPTER 11. NONPARAMETRIC REGRESSION

weight regardless of x: In this sense we can see that the LL estimator is a ‡exible generalization of the linear OLS estimator. Which nonparametric estimator should you use in practice: NW or LL? The theoretical literature shows that neither strictly dominates the other, but we can describe contexts where one or the other does better. Roughly speaking, the NW estimator performs better than the LL estimator when m(x) is close to a ‡at line, but the LL estimator performs better when m(x) is meaningfully non-constant. The LL estimator also performs better for values of x near the boundary of the support of xi :

11.5

Nonparametric Residuals and Regression Fit

The …tted regression at x = xi is m(x b i ) and the …tted residual is m(x b i ):

e^i = yi

As a general rule, but especially when the bandwidth h is small, it is hard to view e^i as a good measure of the …t of the regression. As h ! 0 then m(x b i ) ! yi and therefore e^i ! 0: This clearly indicates over…tting as the true error is not zero. In general, since m(x b i ) is a local average which includes yi ; the …tted value will be necessarily close to yi and the residual e^i small, and the degree of this over…tting increases as h decreases. A standard solution is to measure the …t of the regression at x = xi by re-estimating the model excluding the i’th observation. For Nadaraya-Watson regression, the leave-one-out estimator of m(x) excluding observation i is P

xj

j6=i k

m e i (x) =

x

h xj x h

P

j6=i k

yj :

Notationally, the “ i” subscript is used to indicate that the i’th observation is omitted. The leave-one-out predicted value for yi at x = xi equals P

xj

j6=i k

y~i = m e i (xi ) =

xi h

P

j6=i k

xj

xi

yj :

h

The leave-one-out residuals (or prediction errors) are the di¤erence between the leave-one-out predicted values and the actual observation e~i = yi

y~i :

Since y~i is not a function of yi ; there is no tendency for y~i to over…t for small h: Consequently, e~i is a good measure of the …t of the estimated nonparametric regression. Similarly, the leave-one-out local-linear residual is e~i = yi e i with 0 1 1 X X ei 0 A @ = kij z ij yj ; k z z ij ij ij e i

j6=i

j6=i

z ij =

and kij = k

1

xj

xi

xj

xi h

:

206

CHAPTER 11. NONPARAMETRIC REGRESSION

11.6

Cross-Validation Bandwidth Selection

As we mentioned before, the choice of bandwidth h is crucial. As h increases, the kernel regression estimators (both NW and LL) become more smooth, ironing out the bumps and wiggles. This reduces estimation variance but at the cost of increased bias and oversmoothing. As h decreases the estimators become more wiggly, erratic, and noisy. It is desirable to select h to trade-o¤ these features. How can this be done systematically? To be explicit about the dependence of the estimator on the bandwidth, let us write the estimator of m(x) with a given bandwidth h as m(x; b h); and our discussion will apply equally to the NW and LL estimators. Ideally, we would like to select h to minimize the mean-squared error (MSE) of m(x; b h) as a estimate of m(x): For a given value of x the MSE is M SEn (x; h) = E (m(x; b h)

m(x))2 :

We are typically interested in estimating m(x) for all values in the support of x: A common measure for the average …t is the integrated MSE Z IM SEn (h) = M SEn (x; h)fx (x)dx Z = E (m(x; b h) m(x))2 fx (x)dx

where fx (x) is the marginal density of xi : Notice that we have de…ned the IMSE as an integral with respect to the density fx (x): Other weight functions could be used, but it turns out that this is a convenient choice: The IMSE is closely related with the MSFE of Section 5.7. Let (yn+1 ; xn+1 ) be out-of-sample observations (and thus independent of the sample) and consider predicting yn+1 given xn+1 and the nonparametric estimate m(x; b h): The natural point estimate for yn+1 is m(x b n+1 ; h) which has mean-squared forecast error M SF En (h) = E (yn+1

m(x b n+1 ; h))2

= E (en+1 + m(xn+1 ) = =

m(x b n+1 ; h))2

+ E (m(xn+1 ) m(x b n+1 ; h))2 Z 2 + E (m(x; b h) m(x))2 fx (x)dx 2

where the …nal equality uses the fact that xn+1 is independent of m(x; b h): We thus see that M SF En (h) =

2

+ IM SEn (h):

Since 2 is a constant independent of the bandwidth h; M SF En (h) and IM SEn (h) are equivalent measures of the …t of the nonparameric regression. The optimal bandwidth h is the value which minimizes IM SEn (h) (or equivalently M SF En (h)): While these functions are unknown, we learned in Theorem 5.7.1 that (at least in the case of linear regression) M SF En can be estimated by the sample mean-squared prediction errors. It turns out that this fact extends to nonparametric regression. The nonparametric leave-one-out residuals are e~i (h) = yi

m e i (xi ; h)

where we are being explicit about the dependence on the bandwidth h: The mean squared leaveone-out residuals is n 1X CV (h) = e~i (h)2 : n i=1

207

CHAPTER 11. NONPARAMETRIC REGRESSION This function of h is known as the cross-validation criterion. The cross-validation bandwidth b h is the value which minimizes CV (h) b h = argmin CV (h)

(11.7)

h h`

for some h` > 0: The restriction h h` is imposed so that CV (h) is not evaluated over unreasonably small bandwidths. There is not an explicit solution to the minimization problem (11.7), so it must be solved numerically. A typical practical method is to create a grid of values for h; e.g. [h1 ; h2 ; :::; hJ ], evaluate CV (hj ) for j = 1; :::; J; and set b h=

argmin

CV (h):

h2[h1 ;h2 ;:::;hJ ]

Evaluation using a course grid is typically su¢cient for practical application. Plots of CV (h) against h are a useful diagnostic tool to verify that the minimum of CV (h) has been obtained. We said above that the cross-validation criterion is an estimator of the MSFE. This claim is based on the following result.

Theorem 11.6.1 E (CV (h)) = M SF En

1 (h)

= IM SEn

1 (h)

+

2

(11.8)

Theorem 11.6.1 shows that CV (h) is an unbiased estimator of IM SEn 1 (h) + 2 : The …rst term, IM SEn 1 (h); is the integrated MSE of the nonparametric estimator using a sample of size n 1: If n is large, IM SEn 1 (h) and IM SEn (h) will be nearly identical, so CV (h) is essentially unbiased as an estimator of IM SEn (h) + 2 . Since the second term ( 2 ) is una¤ected by the bandwidth h; it is irrelevant for the problem of selection of h. In this sense we can view CV (h) as an estimator of the IMSE, and more importantly we can view the minimizer of CV (h) as an estimate of the minimizer of IM SEn (h): To illustrate, Figure 11.3 displays the cross-validation criteria CV (h) for the Nadaraya-Watson and Local Linear estimators using the data from Figure 11.1, both using the Epanechnikov kernel. The CV functions are computed on a grid with intervals 0.01. The CV-minimizing bandwidths are h = 1:09 for the Nadaraya-Watson estimator and h = 1:59 for the local linear estimator. Figure 11.3 shows the minimizing bandwidths by the arrows. It is not surprising that the CV criteria recommend a larger bandwidth for the LL estimator than for the NW estimator, as the LL employs more smoothing for a given bandwidth. The CV criterion can also be used to select between di¤erent nonparametric estimators. The CV-selected estimator is the one with the lowest minimized CV criterion. For example, in Figure 11.3, the NW estimator has a minimized CV criterion of 16.88, while the LL estimator has a minimized CV criterion of 16.81. Since the LL estimator achieves a lower value of the CV criterion, LL is the CV-selected estimator. The di¤erence (0.07) is small, suggesting that the two estimators are near equivalent in IMSE. Figure 11.4 displays the …tted CEF estimates (NW and LL) using the bandwidths selected by cross-validation. Also displayed is the true CEF m(x) = 10 ln(x). Notice that the nonparametric estimators with the CV-selected bandwidths (and especially the LL estimator) track the true CEF quite well. Proof of Theorem 11.6.1. Observe that m(xi ) m e i (xi ; h) is a function only of (x1 ; :::; xn ) and (e1 ; :::; en ) excluding ei ; and is thus uncorrelated with ei : Since e~i (h) = m(xi ) m e i (xi ; h) + ei ;

CHAPTER 11. NONPARAMETRIC REGRESSION

208

Figure 11.3: Cross-Validation Criteria, Nadaraya-Watson Regression and Local Linear Regression

Figure 11.4: Nonparametric Estimates using data-dependent (CV) bandwidths

209

CHAPTER 11. NONPARAMETRIC REGRESSION then E (CV (h)) = E e~i (h)2 = E e2i + E (m e i (xi ; h) =

+2E ((m e i (xi ; h) 2

m(xi ))2

m(xi )) ei )

+ E (m e i (xi ; h)

m(xi ))2 :

(11.9)

The second term is an expectation over the random variables xi and m e i (x; h); which are independent as the second is not a function of the i’th observation. Thus taking the conditional expectation given the sample excluding the i’th observation, this is the expectation over xi only, which is the integral with respect to its density Z E i (m e i (xi ; h) m(xi ))2 = (m e i (x; h) m(x))2 fx (x)dx: Taking the unconditional expecation yields E (m e i (xi ; h)

m(xi ))

2

= E

Z

(m e i (x; h)

= IM SEn

m(x))2 fx (x)dx

1 (h)

where this is the IMSE of a sample of size n 1 as the estimator m e Combined with (11.9) we obtain (11.8), as desired.

11.7

i

uses n

1 observations.

Asymptotic Distribution

There is no …nite sample distribution theory for kernel estimators, but there is a well developed asymptotic distribution theory. The theory is based on the approximation that the bandwidth h decreases to zero as the sample size n increases. This means that the smoothing is increasingly localized as the sample size increases. So long as the bandwidth does not decrease to zero too quickly, the estimator can be shown to be asymptotically normal, but with a non-trivial bias. Let fx (x) denote the marginal density of xi and 2 (x) = E e2i j xi = x denote the conditional variance of ei = yi m(xi ):

Theorem 11.7.1 Let m(x) b denote either the Nadarya-Watson or Local Linear estimator of m(x): If x is interior to the support of xi and fx (x) > 0; then as n ! 1 and h ! 0 such that nh ! 1; p

nh m(x) b

m(x)

h2

2 k B(x)

d

! N 0;

Rk 2 (x) fx (x)

(11.10)

where 2k and Rk are de…ned in (11.5) and (11.6). For the NadarayaWatson estimator 1 B(x) = m00 (x) + fx (x) 2

1 0 fx (x)m0 (x)

and for the local linear estimator 1 B(x) = fx (x)m00 (x) 2

210

CHAPTER 11. NONPARAMETRIC REGRESSION

There are several interesting features about the asymptotic distribution which are p noticeably p nh; not n: di¤erent than for parametric estimators. First, the estimator converges at the rate p p Since h ! 0; nh diverges slower than n; thus the nonparametric estimator converges more slowly than a parametric estimator. Second, the asymptotic distribution contains a non-neglible bias term h2 2k B(x): This term asymptotically disappears since h ! 0: Third, the assumptions that nh ! 1 and h ! 0 mean that the estimator is p consistent for the CEF m(x). The fact that the estimator converges at the rate nh has led to the interpretation of nh as the “e¤ective sample size”. This is because the number of observations being used to construct m(x) b is proportional to nh; not n as for a parametric estimator. It is helpful to understand that the nonparametric estimator has a reduced convergence rate because the object being estimated – m(x) – is nonparametric. This is harder than estimating a …nite dimensional parameter, and thus comes at a cost. Unlike parametric estimation, the asymptotic distribution of the nonparametric estimator includes a term representing the bias of the estimator. The asymptotic distribution (11.10) shows the form of this bias. Not only is it proportional to the squared bandwidth h2 (the degree of smoothing), it is proportional to the function B(x) which depends on the slope and curvature of the CEF m(x): Interestingly, when m(x) is constant then B(x) = 0 and the kernel estimator has no asymptotic bias. The bias is essentially increasing in the curvature of the CEF function m(x): This is because the local averaging smooths m(x); and the smoothing induces more bias when m(x) is curved. Theorem 11.7.1 shows that the asymptotic distributions of the NW and LL estimators are similar, with the only di¤erence arising in the bias function B(x): The bias term for the NW estimator has an extra component which depends on the …rst derivative of the CEF m(x) while the bias term of the LL estimator is invariant to the …rst derivative. The fact that the bias formula for the LL estimator is simpler and is free of dependence on the …rst derivative of m(x) suggests that the LL estimator will generally have smaller bias than the NW estimator (but this is not a precise ranking). Since the asymptotic variances in the two distributions are the same, this means that the LL estimator achieves a reduced bias without an e¤ect on asymptotic variance. This analysis has led to the general preference for the LL estimator over the NW estimator in the nonparametrics literature. One implication of Theorem 11.7.1 is that we can de…ne the asymptotic MSE (AMSE) of m(x) b as the squared bias plus the asymptotic variance

Focusing on rates, this says

AM SE (m(x)) b = h2

2 2 k B(x)

AM SE (m(x)) b

h4 +

+

Rk 2 (x) : nhfx (x)

1 nh

(11.11)

(11.12)

which means that the AMSE is dominated by the larger of h4 and (nh) 1 : Notice that the bias is increasing in h and the variance is decreasing in h: (More smoothing means more observations are used for local estimation: this increases the bias but decreases estimation variance.) To select h to minimize the AMSE, these two components should balance each other. Setting h4 / (nh) 1 means setting h / n 1=5 : Another way to see this is to pick h to minimize the right-hand-side of (11.12). The …rst-order condition for h is @ @h

h4 +

1 nh

= 4h3

1 =0 nh2

which when solved for h yields h = n 1=5 : What this means is that for AMSE-e¢cient estimation of m(x); the optimal rate for the bandwidth is h / n 1=5 :

211

CHAPTER 11. NONPARAMETRIC REGRESSION Theorem 11.7.2 The bandwidth which minimizes the AMSE (11.12) is of order h / n 1=5 . With h / n 1=5 then AM SE (m(x)) b = O n 4=5 and m(x) b = m(x) + Op n 2=5 :

This result means that the bandwidth should take the form h = cn 1=5 : The optimal constant c depends on the kernel k; the bias function B(x) and the marginal density fx (x): A common misinterpretation is to set h = n 1=5 ; which is equivalent to setting c = 1 and is completely arbitrary. Instead, an empirical bandwidth selection rule such as cross-validation should be used in practice. When h = cn 1=5 we can rewrite the asymptotic distribution (11.10) as n2=5 (m(x) b

d

m(x)) ! N c2

Rk 2 (x) 2 k B(x); 1=2 c fx (x)

In this representation, we see that m(x) b is asymptotically normal, but with a n2=5 rate of convergence and non-zero mean. The asymptotic distribution depends on the constant c through the bias (positively) and the variance (inversely). The asymptotic distribution in Theorem 11.7.1 allows for the optimal rate h = cn 1=5 but this rate is not required. In particular, consider an undersmoothing (smaller than optimal) bandwith with rate h = o n 1=5 . For example, we could specify that h = cn for some c > 0 and p 1=5 < < 1: Then nhh2 = O(n(1 5 )=2 ) = o(1) so the bias term in (11.10) is asymptotically negligible so Theorem 11.7.1 implies p

nh (m(x) b

d

m(x)) ! N 0;

Rk 2 (x) fx (x)

:

That is, the estimator is asymptotically normal without a bias component. Not having an asymptotic bias component is convenient for some theoretical manipuations, so many authors impose the undersmoothing condition h = o n 1=5 to ensure this situation. This convenience comes at a cost. First, the resulting estimator is ine¢cient as its convergence rate is is Op n (1 )=2 > Op n 2=5 since > 1=5: Second, the distribution theory is an inherently misleading approximation as it misses a critically key ingredient of nonparametric estimation – the trade-o¤ between bias and variance. The approximation (11.10) is superior precisely because it contains the asymptotic bias component which is a realistic implication of nonparametric estimation. Undersmoothing assumptions should be avoided when possible.

11.8

Conditional Variance Estimation

Let’s consider the problem of estimation of the conditional variance 2

(x) = var (yi j xi = x) = E e2i j xi = x :

Even if the conditional mean m(x) is parametrically speci…ed, it is natural to view 2 (x) as inherently nonparametric as economic models rarely specify the form of the conditional variance. Thus it is quite appropriate to estimate 2 (x) nonparametrically. We know that 2 (x) is the CEF of e2i given xi : Therefore if e2i were observed, 2 (x) could be nonparametrically estimated using NW or LL regression. For example, the NW estimator is Pn ki (x)e2i 2 ~ (x) = Pi=1 : n i=1 ki (x)

CHAPTER 11. NONPARAMETRIC REGRESSION

212

Since the errors ei are not observed, we need to replace them with an empirical residual, such as e^i = yi m(x b i ) where m(x) b is the estimated CEF. (The latter could be a nonparametric estimator such as NW or LL, or even a parametric estimator.) Even better, use the leave-one-out predication errors e~i = yi m b i (xi ); as these are not subject to over…tting. With this substitution the NW estimator of the conditional variance is Pn ki (x)~ e2i 2 ^ (x) = Pi=1 : (11.13) n i=1 ki (x)

This estimator depends on a set of bandwidths h1 ; :::; hq , but there is no reason for the bandwidths to be the same as those used to estimate the conditional mean. Cross-validation can be used to select the bandwidths for estimation of ^ 2 (x) separately from cross-validation for estimation of m(x): b There is one subtle di¤erence between CEF and conditional variance estimation. The conditional variance is inherently non-negative 2 (x) 0 and it is desirable for our estimator to satisfy this property. Interestingly, the NW estimator (11.13) is necessarily non-negative, since it is a smoothed average of the non-negative squared residuals, but the LL estimator is not guarenteed to be nonnegative for all x. For this reason, the NW estimator is preferred for conditional variance estimation. Fan and Yao (1998, Biometrika) derive the asymptotic distribution of the estimator (11.13). They obtain the surprising result that the asymptotic distribution of this two-step estimator is identical to that of the one-step idealized estimator ~ 2 (x).

11.9

Standard Errors

Theorem 11.7.1 shows the asymptotic variances of both the NW and LL nonparametric regression estimators equal Rk 2 (x) : V (x) = fx (x) For standard errors we need an estimate of V (x) : A plug-in estimate replaces the unknowns by estimates. The roughness Rk can be found from Table 9.1. The conditional variance can be estimated using (11.13). The density of xi can be estimated using the methods from Section 21.1. Replacing these estimates into the formula for V (x) we obtain the asymptotic variance estimate Rk ^ 2 (x) : V^ (x) = f^x (x) Then an asymptotic standard error for the kernel estimate m(x) b is r 1 ^ s^(x) = V (x): nh Plots of the estimated CEF m(x) b can be accompanied by con…dence intervals m(x) b 2^ s(x): These are known as pointwise con…dence intervals, as they are designed to have correct coverage at each x; not uniformly in x: One important caveat about the interpretation of nonparametric con…dence intervals is that they are not centered at the true CEF m(x); but rather are centered at the biased or pseudo-true value m (x) = m(x) + h2 2k B(x): Consequently, a correct statement about the con…dence interval m(x) b 2^ s(x) is that it asymptotically contains m (x) with probability 95%, not that it asymptotically contains m(x) with probability 95%. The discrepancy is that the con…dence interval does not take into account the bias h2 2k B(x): Unfortunately, nothing constructive can be done about this. The bias is di¢cult and noisy to estimate, so making a bias-correction only in‡ates estimation variance and decreases overall precision.

213

CHAPTER 11. NONPARAMETRIC REGRESSION

A technical “trick” is to assume undersmoothing h = o n 1=5 but this does not really eliminate the bias, it only assumes it away. The plain fact is that once we honestly acknowledge that the true CEF is nonparametric, it then follows that any …nite sample estimate will have …nite sample bias, and this bias will be inherently unknown and thus impossible to incorporate into con…dence intervals.

11.10

Multiple Regressors

Our analysis has focus on the case of real-valued xi for simplicity of exposition, but the methods of kernel regression extend easily to the multiple regressor case, at the cost of a reduced rate of convergence. In this section we consider the case of estimation of the conditional expectation function E (yi j xi = x) = m(x) when

1 x1i C B xi = @ ... A xdi 0

is a d-vector. For any evaluation point x and observation i; de…ne the kernel weights x1i

ki (x) = k

x1 h1

x2i

k

x2 h2

k

xdi

xd hd

;

a d-fold product kernel. The kernel weights ki (x) assess if the regressor vector xi is close to the evaluation point x in the Euclidean space Rd . These weights depend on a set of d bandwidths, hj ; one for each regressor. We can group them together into a single vector for notational convenience: 0 1 h1 B C h = @ ... A : hd

Given these weights, the Nadaraya-Watson estimator takes the form Pn ki (x)yi m(x) b = Pi=1 : n i=1 ki (x)

For the local-linear estimator, de…ne

1

z i (x) =

xi

x

and then the local-linear estimator can be written as m(x) b = b (x) where ! 1 n n X X b (x) 0 = k (x)z (x)z (x) ki (x)z i (x)yi i i i b (x) =

i=1 0

Z KZ

i=1

1

0

Z Ky

where K = diagfk1 (x); :::; kn (x)g: In multiple regressor kernel regression, cross-validation remains a recommended method for bandwidth selection. The leave-one-out residuals e~i and cross-validation criterion CV (h) are de…ned identically as in the single regressor case. The only di¤erence is that now the CV criterion is

214

CHAPTER 11. NONPARAMETRIC REGRESSION

a function over the d-dimensional bandwidth h. This is a critical practical di¤erence since …nding b which minimizes CV (h) can be computationally di¢cult when h is high the bandwidth vector h dimensional. Grid search is cumbersome and costly, since G gridpoints per dimension imply evaulation of CV (h) at Gd distinct points, which can be a large number. Furthermore, plots of CV (h) against h are challenging when d > 2: The asymptotic distribution of the estimators in the multiple regressor case is an extension of the single regressor case. Let fx (x) denote the marginal density of xi and 2 (x) = E e2i j xi = x the conditional variance of ei = yi m(xi ): Let jhj = h1 h2 hd : Theorem 11.10.1 Let m(x) b denote either the Nadarya-Watson or Local Linear estimator of m(x): If x is interior to the support of xi and fx (x) > 0; then as n ! 1 and hj ! 0 such that n jhj ! 1; 1 0 d X p Rkd 2 (x) d 2 2 A b m(x) h B (x) n jhj @m(x) ! N 0; j ;j k fx (x) j=1

where for the Nadaraya-Watson estimator Bj (x) =

1 @2 m(x) + fx (x) 2 @x2j

1

@ @ fx (x) m(x) @xj @xj

and for the Local Linear estimator Bj (x) =

1 @2 m(x) 2 @x2j

For notational simplicity consider the case that there is a single common bandwidth h: In this case the AMSE takes the form 1 AM SE(m(x)) b h4 + nhd That is, the squared bias is of order h4 ; the same as in the single regressor case, but the variance is of larger order (nhd ) 1 : Setting h to balance these two components requires setting h n 1=(4+d) :

Theorem 11.10.2 The bandwidth which minimizes the AMSE is of order h / n 1=(4+d) . With h / n 1=(4+d) then AM SE (m(x)) b = O n 4=(4+d) and m(x) b = m(x) + Op n 2=(4+d) In all estimation problems an increase in the dimension decreases estimation precision. For example, in parametric estimation an increase in dimension typically increases the asymptotic variance. In nonparametric estimation an increase in the dimension typically decreases the convergence rate, which is a more fundamental decrease in precision. For example, in kernel regression the convergence rate Op n 2=(4+d) decreases as d increases. The reason is the estimator m(x) b is a local average of the yi for observations such that xi is close to x, and when there are multiple regressors the number of such observations is inherently smaller. This phenomenon – that the rate of convergence of nonparametric estimation decreases as the dimension increases – is called the curse of dimensionality.

Chapter 12

Series Estimation 12.1

Approximation by Series

As we mentioned at the beginning of Chapter 11, there are two main methods of nonparametric regression: kernel estimation and series estimation. In this chapter we study series methods. Series methods approximate an unknown function (e.g. the CEF m(x)) with a ‡exible parametric function, with the number of parameters treated similarly to the bandwidth in kernel regression. A series approximation to m(x) takes the form mK (x) = mK (x; K ) where mK (x; K ) is a known parametric family and K is an unknown coe¢cient. The integer K is the dimension of K and indexes the complexity of the approximation. A linear series approximation takes the form mK (x) =

K X

zjK (x)

jK

j=1

= z K (x)0

K

where zjK (x) are (nonlinear) functions of x; and are known as basis functions or basis function transformations of x: For real-valued x; a well-known linear series approximation is the p’th-order polynomial mK (x) =

p X

xj

jK

j=0

where K = p + 1: When x 2 Rd is vector-valued, a p’th-order polynomial is mK (x) =

p X

j1 =0

p X

xj11

xjdd

j1 ;:::;jd K :

jd =0

This includes all powers and cross-products, and the coe¢cient vector has dimension K = (p + 1)d : In general, a common method to create a series approximation for vector-valued x is to include all non-redundant cross-products of the basis function transformations of the components of x:

12.2

Splines

Another common series approximation is a continuous piecewise polynomial function known as a spline. While splines can be of any polynomial order (e.g. linear, quadratic, cubic, etc.), a common choice is cubic. To impose smoothness it is common to constrain the spline function to have continuous derivatives up to the order of the spline. Thus a quadratic spline is typically 215

216

CHAPTER 12. SERIES ESTIMATION

constrained to have a continuous …rst derivative, and a cubic spline is typically constrained to have a continuous …rst and second derivative. There is more than one way to de…ne a spline series expansion. All are based on the number of knots – the join points between the polynomial segments. To illustrate, a piecewise linear function with two segments and a knot at t is 8 x t. Its left limit at x = t is 00 and its right limit is 10 ; so is continuous if (and only if) 00 = 10 : Enforcing this constraint is equivalent to writing the function as mK (x) =

0

+

1 (x

mK (x) =

0

t) +

2 (x

t) 1 (x

t)

or after transforming coe¢cients, as +

1x

+

2 (x

t) 1 (x

t)

Notice that this function has K = 3 coe¢cients, the same as a quadratic polynomial. A piecewise quadratic function with one knot at t is 8 2 x 0; let Jn ( ) = n g n ( )0 W n g n ( ): This is a non-negative measure of the “length” of the vector g n ( ): For example, if W n = I; then, Jn ( ) = n g n ( )0 g n ( ) = n kg n ( )k2 ; the square of the Euclidean length. The GMM estimator minimizes Jn ( ).

De…nition 14.2.1 b GM M = argmin Jn ( ) : Note that if k = `; then g n ( b ) = 0; and the GMM estimator is the method of moments estimator: The …rst order conditions for the GMM estimator are @ Jn ( b ) @ @ = 2 g n ( b )0 W n g n ( b ) @ 1 0 1 0 X Z Wn Z y = 2 n n

0 =

so

Xb

2 X 0Z W n Z 0X b = 2 X 0Z W n Z 0y

which establishes the following.

Proposition 14.2.1 b

GM M

=

X 0Z W n Z 0X

1

X 0Z W n Z 0y :

While the estimator depends on W n ; the dependence is only up to scale, for if W n is replaced by cW n for some c > 0; b GM M does not change.

239

CHAPTER 14. GENERALIZED METHOD OF MOMENTS

14.3

Distribution of GMM Estimator p

Assume that W n ! W > 0: Let

Q = E z i x0i

and = E z i z 0i e2i = E g i g 0i ; where g i = z i ei : Then

1 0 X Z Wn n

and

1 0 X Z Wn n

1 0 ZX n

1 p Z 0e n

p

! Q0 W Q

d

! Q0 W N (0; ) :

We conclude: Theorem 14.3.1 Asymptotic Distribution of GMM Estimator p where V

n b

= Q0 W Q

d

! N (0; V ) ;

1

Q0 W W Q

Q0 W Q

1

:

In general, GMM estimators are asymptotically normal with “sandwich form” asymptotic variances. 1 The optimal weight matrix W 0 is one which minimizes V : This turns out to be W 0 = : The proof is left as an exercise. This yields the e¢cient GMM estimator:

Thus we have

b = X 0Z

1

Z 0X

1

X 0Z

1

Z 0 y:

Theorem 14.3.2 Asymptotic Distribution of E¢cient GMM Estimator p 1 d ! N 0; Q0 1 Q : n b p

1 W0 = is not known in practice, but it can be estimated consistently. For any W n ! W 0 ; b we still call the e¢cient GMM estimator, as it has the same asymptotic distribution. By “e¢cient”, we mean that this estimator has the smallest asymptotic variance in the class of GMM estimators with this set of moment conditions. This is a weak concept of optimality, as we are only considering alternative weight matrices W n : However, it turns out that the GMM estimator is semiparametrically e¢cient, as shown by Gary Chamberlain (1987). If it is known that E (g i ( )) = 0; and this is all that is known, this is a semi-parametric problem, as the distribution of the data is unknown. Chamberlain showed that in this context, no semiparametric estimator (one which is consistent globally for the class of models considered) 1 can have a smaller asymptotic variance than G0 1 G where G = E @@ 0 g i ( ): Since the GMM estimator has this asymptotic variance, it is semiparametrically e¢cient. This result shows that in the linear model, no estimator has greater asymptotic e¢ciency than the e¢cient linear GMM estimator. No estimator can do better (in this …rst-order asymptotic sense), without imposing additional assumptions.

240

CHAPTER 14. GENERALIZED METHOD OF MOMENTS

14.4

Estimation of the E¢cient Weight Matrix

Given any weight matrix W n > 0; the GMM estimator b is consistent yet ine¢cient. For 1 example, we can set W n = I ` : In the linear model, a better choice is W n = (Z 0 Z) : Given any such …rst-step estimator, we can de…ne the residuals e^i = yi x0i b and moment equations g ^i = z i e^i = g(yi ; xi ; z i ; b ): Construct n

gn = gn( b ) = g ^i = g ^i

and de…ne

n

Wn =

1X g ^i g ^i 0 n i=1

p

!

1

1X g ^i ; n i=1

gn; n

=

1X g ^i g ^0i n

g n g 0n

i=1

!

1

:

(14.4)

1 Then W n ! = W 0 ; and GMM using W n as the weight matrix is asymptotically e¢cient. A common alternative choice is to set ! 1 n 1X 0 Wn = g ^i g ^i n i=1

which uses the uncentered moment conditions. Since Eg i = 0; these two estimators are asymptotically equivalent under the hypothesis of correct speci…cation. However, Alastair Hall (2000) has shown that the uncentered estimator is a poor choice. When constructing hypothesis tests, under the alternative hypothesis the moment conditions are violated, i.e. Eg i 6= 0; so the uncentered estimator will contain an undesirable bias term and the power of the test will be adversely a¤ected. A simple solution is to use the centered moment conditions to construct the weight matrix, as in (14.4) above. Here is a simple way to compute the e¢cient GMM estimator for the linear model. First, set W n = (Z 0 Z) 1 , estimate b using this weight matrix, and construct the residual e^i = yi x0i b : Then set g ^i = z i e^i ; and let g ^ be the associated n ` matrix: Then the e¢cient GMM estimator is b = X 0Z g ^0 g ^

ng n g 0n

1

Z 0X

1

X 0Z g ^0 g ^

ng n g 0n

1

Z 0 y:

In most cases, when we say “GMM”, we actually mean “e¢cient GMM”. There is little point in using an ine¢cient GMM estimator when the e¢cient estimator is easy to compute. An estimator of the asymptotic variance of ^ can be seen from the above formula. Set Vb = n X 0 Z g ^0 g ^

ng n g 0n

1

Z 0X

1

:

1 Asymptotic standard errors are given by the square roots of the diagonal elements of Vb : n There is an important alternative to the two-step GMM estimator just described. Instead, we can let the weight matrix be considered as a function of : The criterion function is then ! 1 n 1X 0 0 g n ( ): J( ) = n g n ( ) g i ( )g i ( ) n i=1

where gi ( ) = gi( )

gn( )

The b which minimizes this function is called the continuously-updated GMM estimator, and was introduced by L. Hansen, Heaton and Yaron (1996). The estimator appears to have some better properties than traditional GMM, but can be numerically tricky to obtain in some cases. This is a current area of research in econometrics.

241

CHAPTER 14. GENERALIZED METHOD OF MOMENTS

14.5 `

GMM: The General Case

In its most general form, GMM applies whenever an economic or statistical model implies the 1 moment condition E (g i ( )) = 0:

Often, this is all that is known. Identi…cation requires l k = dim( ): The GMM estimator minimizes J( ) = n g n ( )0 W n g n ( ) where

n

gn( ) =

1X gi( ) n i=1

and

n

1X g ^i g ^0i n

Wn =

g n g 0n

i=1

!

1

;

with g ^i = g i ( e ) constructed using a preliminary consistent estimator e , perhaps obtained by …rst setting W n = I: Since the GMM estimator depends upon the …rst-stage estimator, often the weight matrix W n is updated, and then b recomputed. This estimator can be iterated if needed. Theorem 14.5.1 Distribution of Nonlinear GMM Estimator Under general regularity conditions, p where

d

n b

! N 0; G0

1

G

1

;

= E g i g 0i and G=E

@ g ( ): @ 0 i

The variance of b may be estimated by where

Vb

^0 ^ = G

^ =n

1

X

1

^ G

1

^i 0 g ^i g

i

and ^=n G

1

X @ ^ 0 g i ( ): @ i

The general theory of GMM estimation and testing was exposited by L. Hansen (1982).

14.6

Over-Identi…cation Test

Overidenti…ed models (` > k) are special in the sense that there may not be a parameter value such that the moment condition

242

CHAPTER 14. GENERALIZED METHOD OF MOMENTS

Eg(yi ; xi ; z i ; ) = 0 holds. Thus the model – the overidentifying restrictions – are testable. For example, take the linear model yi = 01 x1i + 02 x2i +ei with E (x1i ei ) = 0 and E (x2i ei ) = 0: It is possible that 2 = 0; so that the linear equation may be written as yi = 01 x1i + ei : However, it is possible that 2 6= 0; and in this case it would be impossible to …nd a value of 1 so that both E (x1i (yi x01i 1 )) = 0 and E (x2i (yi x01i 1 )) = 0 hold simultaneously. In this sense an exclusion restriction can be seen as an overidentifying restriction. p Note that g n ! Eg i ; and thus g n can be used to assess whether or not the hypothesis that Eg i = 0 is true or not. The criterion function at the parameter estimates is Jn = n g 0n W n g n = n2 g 0n g ^0 g ^

ng n g 0n

1

gn:

is a quadratic form in g n ; and is thus a natural test statistic for H0 : Eg i = 0:

Theorem 14.6.1 (Sargan-Hansen). Under the hypothesis of correct speci…cation, and if the weight matrix is asymptotically e¢cient, d

Jn = Jn ( b ) !

2 ` k:

The proof of the theorem is left as an exercise. This result was established by Sargan (1958) for a specialized case, and by L. Hansen (1982) for the general case. The degrees of freedom of the asymptotic distribution are the number of overidentifying restrictions. If the statistic J exceeds the chi-square critical value, we can reject the model. Based on this information alone, it is unclear what is wrong, but it is typically cause for concern. The GMM overidenti…cation test is a very useful by-product of the GMM methodology, and it is advisable to report the statistic J whenever GMM is the estimation method. When over-identi…ed models are estimated by GMM, it is customary to report the J statistic as a general test of model adequacy.

14.7

Hypothesis Testing: The Distance Statistic

We described before how to construct estimates of the asymptotic covariance matrix of the GMM estimates. These may be used to construct Wald tests of statistical hypotheses. If the hypothesis is non-linear, a better approach is to directly use the GMM criterion function. This is sometimes called the GMM Distance statistic, and sometimes called a LR-like statistic (the LR is for likelihood-ratio). The idea was …rst put forward by Newey and West (1987). For a given weight matrix W n ; the GMM criterion function is Jn ( ) = n g n ( )0 W n g n ( ) For h : Rk ! Rr ; the hypothesis is H0 : h( ) = 0: The estimates under H1 are

b = argmin Jn ( )

CHAPTER 14. GENERALIZED METHOD OF MOMENTS and those under H0 are

243

e = argmin J( ): h( )=0

The two minimizing criterion functions are Jn ( b ) and Jn ( e ): The GMM distance statistic is the di¤erence Dn = Jn ( e ) Jn ( b ): Proposition 14.7.1 If the same weight matrix W n is used for both null and alternative, 1. D

0 d

2. D !

2 r

3. If h is linear in

; then D equals the Wald statistic.

If h is non-linear, the Wald statistic can work quite poorly. In contrast, current evidence suggests that the Dn statistic appears to have quite good sampling properties, and is the preferred test statistic. Newey and West (1987) suggested to use the same weight matrix W n for both null and alternative, as this ensures that Dn 0. This reasoning is not compelling, however, and some current research suggests that this restriction is not necessary for good performance of the test. This test shares the useful feature of LR tests in that it is a natural by-product of the computation of alternative models.

14.8

Conditional Moment Restrictions

In many contexts, the model implies more than an unconditional moment restriction of the form Eg i ( ) = 0: It implies a conditional moment restriction of the form E (ei ( ) j z i ) = 0 where ei ( ) is some s 1 function of the observation and the parameters. In many cases, s = 1. It turns out that this conditional moment restriction is much more powerful, and restrictive, than the unconditional moment restriction discussed above. Our linear model yi = x0i + ei with instruments z i falls into this class under the stronger assumption E (ei j z i ) = 0: Then ei ( ) = yi x0i : It is also helpful to realize that conventional regression models also fall into this class, except that in this case xi = z i : For example, in linear regression, ei ( ) = yi x0i , while in a nonlinear regression model ei ( ) = yi g(xi ; ): In a joint model of the conditional mean and variance 8 yi x0i < : ei ( ; ) = : 2 0 0 (yi xi ) f (xi ) Here s = 2: Given a conditional moment restriction, an unconditional moment restriction can always be constructed. That is for any ` 1 function (xi ; ) ; we can set g i ( ) = (xi ; ) ei ( ) which satis…es Eg i ( ) = 0 and hence de…nes a GMM estimator. The obvious problem is that the class of functions is in…nite. Which should be selected?

CHAPTER 14. GENERALIZED METHOD OF MOMENTS

244

This is equivalent to the problem of selection of the best instruments. If xi 2 R is a valid instrument satisfying E (ei j xi ) = 0; then xi ; x2i ; x3i ; :::; etc., are all valid instruments. Which should be used? One solution is to construct an in…nite list of potent instruments, and then use the …rst k instruments. How is k to be determined? This is an area of theory still under development. A recent study of this problem is Donald and Newey (2001). Another approach is to construct the optimal instrument. The form was uncovered by Chamberlain (1987). Take the case s = 1: Let @ ei ( ) j z i @

Ri = E and 2 i

= E ei ( )2 j z i :

Then the “optimal instrument” is Ai =

i

2

Ri

so the optimal moment is g i ( ) = Ai ei ( ): Setting g i ( ) to be this choice (which is k 1; so is just-identi…ed) yields the best GMM estimator possible. In practice, Ai is unknown, but its form does help us think about construction of optimal instruments. In the linear model ei ( ) = yi x0i ; note that E (xi j z i )

Ri = and 2 i

= E e2i j z i ;

so Ai =

i

2

E (xi j z i ) :

In the case of linear regression, xi = z i ; so Ai = i 2 z i : Hence e¢cient GMM is GLS, as we discussed earlier in the course. In the case of endogenous variables, note that the e¢cient instrument Ai involves the estimation of the conditional mean of xi given z i : In other words, to get the best instrument for xi ; we need the best conditional mean model for xi given z i , not just an arbitrary linear projection. The e¢cient instrument is also inversely proportional to the conditional variance of ei : This is the same as the GLS estimator; namely that improved e¢ciency can be obtained if the observations are weighted inversely to the conditional variance of the errors.

14.9

Bootstrap GMM Inference

Let b be the 2SLS or GMM estimator of . Using the EDF of (yi ; z i ; xi ), we can apply the bootstrap methods discussed in Chapter 10 to compute estimates of the bias and variance of b ; and construct con…dence intervals for ; identically as in the regression model. However, caution should be applied when interpreting such results. A straightforward application of the nonparametric bootstrap works in the sense of consistently achieving the …rst-order asymptotic distribution. This has been shown by Hahn (1996). However, it fails to achieve an asymptotic re…nement when the model is over-identi…ed, jeopardizing the theoretical justi…cation for percentile-t methods. Furthermore, the bootstrap applied J test will yield the wrong answer.

245

CHAPTER 14. GENERALIZED METHOD OF MOMENTS

The problem is that in the sample, b is the “true” value and yet g n ( ^ ) 6= 0: Thus according to random variables (yi ; z i ; xi ) drawn from the EDF Fn ; E gi b

= g n ( ^ ) 6= 0:

This means that (yi ; z i ; xi ) do not satisfy the same moment conditions as the population distribution. A correction suggested by Hall and Horowitz (1996) can solve the problem. Given the bootstrap sample (y ; Z ; X ); de…ne the bootstrap GMM criterion Jn ( ) = n

gn( )

gn( ^ )

0

W n gn( )

gn( ^ )

where g n ( b ) is from the in-sample data, not from the bootstrap data. Let b minimize Jn ( ); and de…ne all statistics and tests accordingly. In the linear model, this implies that the bootstrap estimator is b = X 0Z W Z 0X n n

1

X 0Z W n Z 0y

Z 0e ^

:

where e ^ = y X b are the in-sample residuals. The bootstrap J statistic is Jn ( b ): Brown and Newey (2002) have an alternative solution. They note that we can sample from the observations with the empirical likelihood probabilities p^i described in Chapter 15. Since Pn ^i g i b = 0; this sampling scheme preserves the moment conditions of the model, so no i=1 p recentering or adjustments is needed. Brown and Newey argue that this bootstrap procedure will be more e¢cient than the Hall-Horowitz GMM bootstrap.

246

CHAPTER 14. GENERALIZED METHOD OF MOMENTS

Exercises Exercise 14.1 Take the model yi = x0i + ei E (xi ei ) = 0 e2i = z 0i +

i

E (z i i ) = 0: Find the method of moments estimators ^ ; ^ for ( ; ) : Exercise 14.2 Take the single equation y = X +e E (e j Z) = 0 Assume E e2i j z i = then

2:

Show that if ^ is estimated by GMM with weight matrix W n = (Z 0 Z) p

d

n b

! N 0;

2

Q0 M

1

Q

1

;

1

where Q = E (z i x0i ) and M = E (z i z 0i ) : Exercise 14.3 Take the model yi = x0i + ei with E (z i ei ) = 0: Let e^i = yi x0i b where b is consistent for (e.g. a GMM estimator with arbitrary weight matrix). De…ne the estimate of the optimal GMM weight matrix ! 1 n 1X : Wn = z i z 0i e^2i n i=1

p

Show that W n !

1

where

= E z i z 0i e2i :

Exercise 14.4 In the linear model estimated by GMM with general weight matrix W ; the asymptotic variance of ^ GM M is V = Q0 W Q

1

1

(a) Let V 0 be this matrix when W =

Q0 W W Q Q0 W Q

: Show that V 0 = Q0

1

1

Q

1

:

(b) We want to show that for any W ; V V 0 is positive semi-de…nite (for then V 0 is the smaller 1 possible covariance matrix and W = is the e¢cient weight matrix). To do this, start by …nding matrices A and B such that V = A0 A and V 0 = B 0 B: (c) Show that B 0 A = B 0 B and therefore that B 0 (d) Use the expressions V = A0 A; A = B + (A V V 0:

(A

B) = 0:

B) ; and B 0

(A

B) = 0 to show that

Exercise 14.5 The equation of interest is yi = m(xi ; ) + ei E (z i ei ) = 0: The observed data is (yi ; z i ; xi ). z i is ` GMM estimator for .

1 and

is k

1; `

k: Show how to construct an e¢cient

247

CHAPTER 14. GENERALIZED METHOD OF MOMENTS

Exercise 14.6 In the linear model y = X + e with E(xi ei ) = 0; a Generalized Method of Moments (GMM) criterion function for is de…ned as 1 (y n

Jn ( ) =

1

X )0 X b

X 0 (y

X )

(14.5)

P where b = n1 ni=1 xi x0i e^2i ; e^i = yi x0i ^ are the OLS residuals, and b = (X 0 X) GMM estimator of ; subject to the restriction h( ) = 0; is de…ned as

1

X 0 y is LS: The

e = argmin Jn ( ): h( )=0

The GMM test statistic (the distance statistic) of the hypothesis h( ) = 0 is D = Jn ( e ) = min Jn ( ):

(14.6)

h( )=0

(a) Show that you can rewrite Jn ( ) in (14.5) as

0

b

Jn ( ) = n

1

V^

b

thus e is the same as the minimum distance estimator.

(b) Show that in this setting, the distance statistic D in (14.6) equals the Wald statistic. Exercise 14.7 Take the linear model yi = x0i + ei E (z i ei ) = 0: and consider the GMM estimator ^ of

: Let

Jn = ng n ( b )0 b

1

gn( b )

d

denote the test of overidentifying restrictions. Show that Jn ! each of the following: (a) Since

1

> 0; we can write

(b) Jn = n C 0 g n ( b )

0

C0 ^ C

(c) C 0 g n ( b ) = D n C 0 g n (

0)

gn(

0)

p

(d) D n ! I ` (e) n1=2 C 0 g n ( d

R (R0 R)

(f) Jn ! u0 I ` (g) u0 I ` Hint: I `

= C0

d

!u

1

1 0 ZX n

1 0 XZ n

R (R0 R)

1

R (R0 R)

N (0; I ` )

R0 u 1

b

R0 where R = C 0 E (z i x0i )

R (R0 R)

1

1

C

as n ! 1 by demonstrating

1

C 0gn( b )

1 0 Z e: n

=

0)

where

C0

Dn = I `

1

= CC 0 and

2 ` k

R0 u 2 : ` k

R0 is a projection matrix.

1

1 0 ZX n

1

1 0 XZ n

b

1

C0

1

Chapter 15

Empirical Likelihood 15.1

Non-Parametric Likelihood

An alternative to GMM is empirical likelihood. The idea is due to Art Owen (1988, 2001) and has been extended to moment condition models by Qin and Lawless (1994). It is a non-parametric analog of likelihood estimation. The idea is to construct a multinomial distribution F (p1 ; :::; pn ) which places probability pi at each observation. To be a valid multinomial distribution, these probabilities must satisfy the requirements that pi 0 and n X pi = 1: (15.1) i=1

Since each observation is observed once in the sample, the log-likelihood function for this multinomial distribution is n X log L (p1 ; :::; pn ) = log(pi ): (15.2) i=1

First let us consider a just-identi…ed model. In this case the moment condition places no additional restrictions on the multinomial distribution. The maximum likelihood estimators of the probabilities (p1 ; :::; pn ) are those which maximize the log-likelihood subject to the constraint (15.1). This is equivalent to maximizing ! n n X X log(pi ) pi 1 i=1

i=1

where is a Lagrange multiplier. The n …rst order conditions are 0 = pi 1 : Combined with the 1 constraint (15.1) we …nd that the MLE is pi = n yielding the log-likelihood n log(n): Now consider the case of an overidenti…ed model with moment condition Eg i (

0)

=0

where g is ` 1 and is k 1 and for simplicity we write g i ( ) = g(yi ; z i ; xi ; ): The multinomial distribution which places probability pi at each observation (yi ; xi ; z i ) will satisfy this condition if and only if n X pi g i ( ) = 0 (15.3) i=1

The empirical likelihood estimator is the value of which maximizes the multinomial loglikelihood (15.2) subject to the restrictions (15.1) and (15.3).

248

249

CHAPTER 15. EMPIRICAL LIKELIHOOD The Lagrangian for this maximization problem is L ( ; p1 ; :::; pn ; ; ) = where are

and

n X

n X

log(pi )

i=1

pi

1

i=1

!

n

0

n X

pi g i ( )

i=1

are Lagrange multipliers. The …rst-order-conditions of L with respect to pi , ; and 1 pi n X

+ n 0gi ( )

=

pi = 1

i=1

n X

pi g i ( ) = 0:

i=1

Multiplying the …rst equation by pi , summing over i; and using the second and third equations, we …nd = n and 1 : pi = n 1 + 0gi ( ) Substituting into L we …nd R( ; ) =

n log (n)

n X

log 1 +

0

gi ( ) :

(15.4)

i=1

For given

; the Lagrange multiplier

( ) minimizes R ( ; ) : ( ) = argmin R( ; ):

(15.5)

This minimization problem is the dual of the constrained maximization problem. The solution (when it exists) is well de…ned since R( ; ) is a convex function of : The solution cannot be obtained explicitly, but must be obtained numerically (see section 6.5). This yields the (pro…le) empirical log-likelihood function for . R( ) = R( ; ( )) =

n log (n)

n X

log 1 + ( )0 g i ( )

i=1

The EL estimate ^ is the value which maximizes R( ); or equivalently minimizes its negative ^ = argmin [ R( )]

(15.6)

Numerical methods are required for calculation of ^ (see Section 15.5). As a by-product of estimation, we also obtain the Lagrange multiplier ^ = ( ^ ); probabilities p^i =

1 0 n 1 + ^ gi ^

:

and maximized empirical likelihood R( ^ ) =

n X i=1

log (^ pi ) :

(15.7)

250

CHAPTER 15. EMPIRICAL LIKELIHOOD

15.2 Let

Asymptotic Distribution of EL Estimator 0

denote the true value of

and de…ne Gi ( ) =

@ g ( ) @ 0 i

G = EGi ( = E gi (

0)

0 0) gi ( 0)

and V = G0 V

1

For example, in the linear model, Gi ( ) =

1

G

G G0

=

(15.8)

1

(15.9) 1

G

z i x0i ; G =

G0

(15.10)

E (z i x0i ), and

= E z i z 0i e2i :

Theorem 15.2.1 Under regularity conditions, p

d

n ^ 0 p d n^ !

! N (0; V )

1

N (0; V )

where V and V are de…ned in (15.9) and (15.10), and p ^ n are asymptotically independent.

p

n ^

0

and

The theorem shows that the asymptotic variance V for ^ is the same as for e¢cient GMM. Thus the EL estimator is asymptotically e¢cient. Chamberlain (1987) showed that V is the semiparametric e¢ciency bound for in the overidenti…ed moment condition model. This means that no consistent estimator for this class of models can have a lower asymptotic variance than V . Since the EL estimator achieves this bound, it is an asymptotically e¢cient estimator for . Proof of Theorem 15.2.1. ( ^ ; ^ ) jointly solve

0 =

0 =

gi ^

@ R( ; ) = @

n X

@ R( ; ) = @

n Gi ^ X i=1

(15.11)

0 1 + ^ gi ^

i=1

0

0 1 + ^ gi ^

:

P P P Let Gn = n1 ni=1 Gi ( 0 ) ; g n = n1 ni=1 g i ( 0 ) and n = n1 ni=1 g i ( Expanding (15.12) around = 0 and = 0 = 0 yields 0 ' G0n ^ Expanding (15.11) around

=

0

and

0'

= gn

0

0

:

(15.12) 0 0) gi ( 0) :

(15.13)

= 0 yields

Gn ^

0

+

n

^

(15.14)

251

CHAPTER 15. EMPIRICAL LIKELIHOOD Premultiplying by G0n

n

1

and using (15.13) yields

0 '

G0n

n

=

G0n

n

1

gn

G0n

n

1

gn

G0n

n

1

Gn ^

0

1

Gn ^

0

Solving for ^ and using the WLLN and CLT yields p G0n n 1 Gn n ^ 0 ' d

! G0

1

1 1

G

+ G0n

G0n

n

G0

1

1p

n

1

n

^

ng n

(15.15)

N (0; )

= N (0; V ) Solving (15.14) for ^ and using (15.15) yields p n ^ ' n 1 I Gn G0n d

!

=

1 1

n

G G0

I

1

1

Gn

1

1

G

G0n

G0

n

p

1

1

ng n

(15.16)

N (0; )

N (0; V )

Furthermore, since G0 I p

n ^

15.3

0

and

1

G G0

1

1

G

G0 = 0

p ^ n are asymptotically uncorrelated and hence independent.

Overidentifying Restrictions

In a parametric likelihood context, tests are based on the di¤erence in the log likelihood functions. The same statistic can be constructed for empirical likelihood. Twice the di¤erence between the unrestricted empirical log-likelihood n log (n) and the maximized empirical log-likelihood for the model (15.7) is n X 0 LRn = 2 log 1 + ^ g i ^ : (15.17) i=1

Theorem 15.3.1 If Eg i (

d

2 : ` k

= 0 then LRn !

0)

The EL overidenti…cation test is similar to the GMM overidenti…cation test. They are asymptotically …rst-order equivalent, and have the same interpretation. The overidenti…cation test is a very useful by-product of EL estimation, and it is advisable to report the statistic LRn whenever EL is the estimation method. Proof of Theorem 15.3.1. First, by a Taylor expansion, (15.15), and (15.16), n

1 X p gi ^ n i=1

'

p

n g n + Gn ^

'

I

'

n

p

Gn G0n n^:

n

1

0

Gn

1

G0n

n

1

p

ng n

252

CHAPTER 15. EMPIRICAL LIKELIHOOD Second, since log(1 + u) ' u

u2 =2 for u small,

LRn =

n X

0 2 log 1 + ^ g i ^

i=1

' 2^

n 0X i=1

0 ' n^

n X

gi ^ gi ^

0

^

i=1

^ n

d

! N (0; V )0

=

^0

gi ^

1

N (0; V )

2 ` k

where the proof of the …nal equality is left as an exercise.

15.4

Testing

Let the maintained model be Eg i ( ) = 0

(15.18)

where g is ` 1 and is k 1: By “maintained” we mean that the overidentfying restrictions contained in (15.18) are assumed to hold and are not being challenged (at least for the test discussed in this section). The hypothesis of interest is h( ) = 0: where h : Rk ! Ra : The restricted EL estimator and likelihood are the values which solve ~ = argmax R( ) h( )=0

R( ~ ) =

max R( ):

h( )=0

Fundamentally, the restricted EL estimator ~ is simply an EL estimator with ` k+a overidentifying restrictions, so there is no fundamental change in the distribution theory for ~ relative to ^ : To test the hypothesis h( ) while maintaining (15.18), the simple overidentifying restrictions test (15.17) is not appropriate. Instead we use the di¤erence in log-likelihoods: LRn = 2 R( ^ )

R( ~ ) :

This test statistic is a natural analog of the GMM distance statistic.

d

Theorem 15.4.1 Under (15.18) and H0 : h( ) = 0; LRn !

The proof of this result is more challenging and is omitted.

2: a

253

CHAPTER 15. EMPIRICAL LIKELIHOOD

15.5

Numerical Computation

Gauss code which implements the methods discussed below can be found at http://www.ssc.wisc.edu/~bhansen/progs/elike.prc Derivatives The numerical calculations depend on derivatives of the dual likelihood function (15.4). De…ne gi ( ) 1 + 0gi ( )

gi ( ; ) = Gi ( ; ) =

Gi ( )0 1 + 0gi ( )

The …rst derivatives of (15.4) are @ R( ; ) = @

=

R

@ R( ; ) = @

=

R

n X

gi ( ; )

i=1 n X

Gi ( ; ) :

i=1

The second derivatives are R R

R

= =

=

@2 @ @ @2 @ @ @2 @ @

0R(

; )=

n X

g i ( ; ) g i ( ; )0

i=1

0R(

; )=

n X i=1

0R(

; )=

n X i=1

g i ( ; ) Gi ( ; )0 0

@Gi ( ; ) Gi ( ; )0

Gi ( ) 1 + 0gi ( ) @2 @ @

0

1+

g i ( )0 0

gi ( )

1 A

Inner Loop The so-called “inner loop” solves (15.5) for given : The modi…ed Newton method takes a quadratic approximation to Rn ( ; ) yielding the iteration rule j+1

=

j

(R

( ;

j ))

1

R ( ;

j) :

(15.19)

where > 0 is a scalar steplength (to be discussed next). The starting value 1 can be set to the zero vector. The iteration (15.19) is continued until the gradient R ( ; j ) is smaller than some prespeci…ed tolerance. E¢cient convergence requires a good choice of steplength : One method uses the following quadratic approximation. Set 0 = 0; 1 = 21 and 2 = 1: For p = 0; 1; 2; set p

=

j

Rp = R ( ;

p (R

( ;

j ))

1

R ( ;

j ))

p)

A quadratic function can be …t exactly through these three points. The value of this quadratic is ^ = R2 + 3R0 4R1 : 4R2 + 4R0 8R1 yielding the steplength to be plugged into (15.19).

which minimizes

254

CHAPTER 15. EMPIRICAL LIKELIHOOD A complication is that

must be constrained so that 0 n 1+

0

gi ( )

pi

1 which holds if

1

(15.20)

for all i: If (15.20) fails, the stepsize needs to be decreased. Outer Loop The outer loop is the minimization (15.6). This can be done by the modi…ed Newton method described in the previous section. The gradient for (15.6) is R = since R ( ; ) = 0 at

@ @ R( ) = R( ; ) = R + @ @

0

R =R

= ( ); where =

@ ( )= @ 0

1

R

R

;

the second equality following from the implicit function theorem applied to R ( ; ( )) = 0: The Hessian for (15.6) is R

=

@ R( ) @ @ 0 @ R ( ; ( )) + @ 0 R ( ; ( )) + R0

= R

0

= =

R

1

R

0

R ( ; ( )) +

0

R

It is not guaranteed that R > 0: If not, the eigenvalues of R are positive. The Newton iteration rule is

where

=

0

R

:

R

j+1

+

j

R

1

should be adjusted so that all

R

is a scalar stepsize, and the rule is iterated until convergence.

Chapter 16

Endogeneity We say that there is endogeneity in the linear model y = x0i + ei if is the parameter of interest and E(xi ei ) 6= 0: This cannot happen if is de…ned by linear projection, so requires a structural interpretation. The coe¢cient must have meaning separately from the de…nition of a conditional mean or linear projection. Example: Measurement error in the regressor. Suppose that (yi ; xi ) are joint random variables, E(yi j xi ) = xi 0 is linear, is the parameter of interest, and xi is not observed. Instead we observe xi = xi + ui where ui is an k 1 measurement error, independent of yi and xi : Then yi = xi 0 + ei ui )0

= (xi =

x0i

+ ei

+ vi

where u0i :

vi = ei The problem is that

u0i

E (xi vi ) = E (xi + ui ) ei if

=

E ui u0i

6= 0

6= 0 and E (ui u0i ) 6= 0: It follows that if ^ is the OLS estimator, then ^ p!

E xi x0i

=

1

E ui u0i

6= :

This is called measurement error bias. Example: Supply and Demand. The variables qi and pi (quantity and price) are determined jointly by the demand equation qi = 1 pi + e1i and the supply equation qi =

2 pi

+ e2i :

e1i is iid, Eei = 0; 1 + 2 = 1 and Eei e0i = I 2 (the latter for simplicity). e2i The question is, if we regress qi on pi ; what happens? It is helpful to solve for qi and pi in terms of the errors. In matrix notation, Assume that ei =

1 1

1 2

qi pi

255

=

e1i e2i

256

CHAPTER 16. ENDOGENEITY so qi pi

1

1 1

=

2 2

=

e1i e2i

1

1

1 2 e1i

=

e1i e2i

1

+

(e1i

1 e2i e2i )

:

The projection of qi on pi yields qi =

pi + " i

E (pi "i ) = 0 where = Hence if it is estimated by OLS, ^ simultaneous equations bias.

16.1

E (pi qi ) = E p2i

p

!

2

1

2

; which does not equal either

1

or

2:

This is called

Instrumental Variables

Let the equation of interest be yi = x0i + ei

(16.1)

where xi is k 1; and assume that E(xi ei ) 6= 0 so there is endogeneity. We call (16.1) the structural equation. In matrix notation, this can be written as y = X + e:

(16.2)

Any solution to the problem of endogeneity requires additional information which we call instruments. De…nition 16.1.1 The ` 1 random vector z i is an instrumental variable for (16.1) if E (z i ei ) = 0:

In a typical set-up, some regressors in xi will be uncorrelated with ei (for example, at least the intercept). Thus we make the partition xi =

x1i x2i

k1 k2

(16.3)

where E(x1i ei ) = 0 yet E(x2i ei ) 6= 0: We call x1i exogenous and x2i endogenous. By the above de…nition, x1i is an instrumental variable for (16.1); so should be included in z i : So we have the partition k1 x1i (16.4) zi = `2 z 2i where x1i = z 1i are the included exogenous variables, and z 2i are the excluded exogenous variables. That is z 2i are variables which could be included in the equation for yi (in the sense that they are uncorrelated with ei ) yet can be excluded, as they would have true zero coe¢cients in the equation. The model is just-identi…ed if ` = k (i.e., if `2 = k2 ) and over-identi…ed if ` > k (i.e., if `2 > k2 ): We have noted that any solution to the problem of endogeneity requires instruments. This does not mean that valid instruments actually exist.

257

CHAPTER 16. ENDOGENEITY

16.2

Reduced Form

The reduced form relationship between the variables or “regressors” xi and the instruments z i is found by linear projection. Let 1

= E z i z 0i be the `

E z i x0i

k matrix of coe¢cients from a projection of xi on z i ; and de…ne 0

ui = xi

zi

as the projection error. Then the reduced form linear relationship between xi and z i is 0

xi =

z i + ui :

(16.5)

In matrix notation, we can write (16.5) as X =Z +U

(16.6)

where U is n k: By construction, E(z i u0i ) = 0; so (16.5) is a projection and can be estimated by OLS: 0 xi = b z i + u ^i :

or where

b X = Zb + U

b = Z 0Z

1

Z 0X :

Substituting (16:6) into (16.2), we …nd

y = (Z + U )

+e

= Z + v;

(16.7)

where =

(16.8)

and v=U

+ e:

Observe that E (z i vi ) = E z i u0i

+ E (z i ei ) = 0:

Thus (16.7) is a projection equation and may be estimated by OLS. This is y = Z^ + v ^; 0 ^ = Z Z 1 Z 0y The equation (16.7) is the reduced form for y: (16.6) and (16.7) together are the reduced form equations for the system y = Z +v X = Z + U: As we showed above, OLS yields the reduced-form estimates ^ ; ^

258

CHAPTER 16. ENDOGENEITY

16.3

Identi…cation

The structural parameter relates to ( ; ) through (16.8). The parameter meaning that it can be recovered from the reduced form, if rank ( ) = k:

is identi…ed, (16.9)

1 Assume that (16.9) holds. If ` = k; then = : If ` > k; then for any W > 0; = 1 0 0 ( W ) W : If (16.9) is not satis…ed, then cannot be recovered from ( ; ) : Note that a necessary (although not su¢cient) condition for (16.9) is ` k: Since Z and X have the common variables X 1 ; we can rewrite some of the expressions. Using (16.3) and (16.4) to make the matrix partitions Z = [Z 1 ; Z 2 ] and X = [Z 1 ; X 2 ] ; we can partition as

=

11

12

21

22

I 0

=

12 22

(16.6) can be rewritten as X1 = Z1 X2 = Z1

12

+ Z2

22

+ U 2:

(16.10)

is identi…ed if rank( ) = k; which is true if and only if rank( 22 ) = k2 (by the upper-diagonal structure of ): Thus the key to identi…cation of the model rests on the `2 k2 matrix 22 in (16.10).

16.4

Estimation

The model can be written as yi = x0i + ei E (z i ei ) = 0 or Eg i ( ) = 0 x0i

g i ( ) = z i yi

:

This is a moment condition model. Appropriate estimators include GMM and EL. The estimators and distribution theory developed in those Chapter 8 and 9 directly apply. Recall that the GMM estimator, for given weight matrix W n ; is 1

^ = X 0 ZW n Z 0 X

16.5

X 0 ZW n Z 0 y:

Special Cases: IV and 2SLS

If the model is just-identi…ed, so that k = `; then the formula for GMM simpli…es. We …nd that b =

X 0 ZW n Z 0 X

1

X 0 ZW n Z 0 y

=

Z 0X

1

W n 1 X 0Z

=

Z 0X

1

Z 0y

1

X 0 ZW n Z 0 y

259

CHAPTER 16. ENDOGENEITY

This estimator is often called the instrumental variables estimator (IV) of ; where Z is used as an instrument for X: Observe that the weight matrix W n has disappeared. In the just-identi…ed case, the weight matrix places no role. This is also the method of moments estimator of ; and the 1 EL estimator. Another interpretation stems from the fact that since = ; we can construct the Indirect Least Squares (ILS) estimator: b = b =

1

b

1

Z 0Z

1

Z 0X

=

Z 0X

1

Z 0Z

=

Z 0X

1

Z 0y :

Z 0Z 1

Z 0Z

1

Z 0y

Z 0y

which again is the IV estimator. Recall that the optimal weight matrix is an estimate of the inverse of = E z i z 0i e2i : In the 2 2 0 2 / E (z i z 0i ) suggesting (homoskedasticity), then = E (z i z i ) special case that E ei j z i = 1 the weight matrix W n = (Z 0 Z) : Using this choice, the GMM estimator equals b

2SLS

1

= X 0Z Z 0Z

1

Z 0X

X 0Z Z 0Z

1

Z 0y

This is called the two-stage-least squares (2SLS) estimator. It was originally proposed by Theil (1953) and Basmann (1957), and is the classic estimator for linear equations with instruments. Under the homoskedasticity assumption, the 2SLS estimator is e¢cient GMM, but otherwise it is ine¢cient. It is useful to observe that writing 1

then the 2SLS estimator is

P = Z Z 0Z Z0 c = PX = Zb X b =

1

X 0P X c0 X c X

=

1

X 0P y

c0 y: X

The source of the “two-stage” name is since it can be computed as follows First regress X on Z; vis., b = (Z 0 Z)

1

c c0 X c vis., b = X Second, regress y on X;

c = Z b = P X: (Z 0 X) and X 1

c0 y: X

c Recall, X = [X 1 ; X 2 ] and Z = [X 1 ; Z 2 ]: Then It is useful to scrutinize the projection X: i h c1 ; X c2 c = X X = [P X 1 ; P X 2 ]

= [X 1 ; P X 2 ] h i c2 ; = X 1; X

c2 : So only the since X 1 lies in the span of Z: Thus in the second stage, we regress y on X 1 and X endogenous variables X 2 are replaced by their …tted values: c2 = Z 1 b 12 + Z 2 b 22 : X

260

CHAPTER 16. ENDOGENEITY

16.6

Bekker Asymptotics

Bekker (1994) used an alternative asymptotic framework to analyze the …nite-sample bias in the 2SLS estimator. Here we present a simpli…ed version of one of his results. In our notation, the model is y = X +e

(16.11)

X = Z +U

(16.12)

= (e; U ) E

E ( j Z) = 0 0

jZ

= S

As before, Z is n l so there are l instruments. First, let’s analyze the approximate bias of OLS applied to (16.11). Using (16.12), 1 0 Xe n

E

= E (xi ei ) =

0

E (z i ei ) + E (ui ei ) = s21

and E

1 0 XX n

= E xi x0i + E ui z 0i

=

0

E z i z 0i

=

0

Q + S 22

0

+

E z i u0i + E ui u0i

where Q = E (z i z 0i ) : Hence by a …rst-order approximation E ^ OLS

0

=

1

1 0 XX n

E

Q + S 22

E 1

1 0 Xe n

s21

(16.13)

which is zero only when s21 = 0 (when X is exogenous). We now derive a similar result for the 2SLS estimator. ^ 2SLS = X 0 P X

1

X 0P y :

1

Let P = Z (Z 0 Z) Z 0 . By the spectral decomposition of an idempotent matrix, P = H H 0 where = diag (I l ; 0) : Let Q = H 0 S 1=2 which satis…es EQ0 Q = I n and partition Q = (q 01 Q02 ) where q 1 is l 1: Hence E

1 0 P jZ n

1 1=20 S E Q0 Q j Z S 1=2 n 1 0 1 1=20 S E q q S 1=2 n n 1 1 l 1=20 1=2 S S n S

= = = =

where =

l : n

Using (16.12) and this result, 1 1 E X 0P e = E n n

0

Z 0e +

1 E U 0 P e = s21 ; n

261

CHAPTER 16. ENDOGENEITY and 1 E X 0P X n

=

0

E z i z 0i

=

0

Q + S 22 :

0

+

E (z i ui ) + E ui z 0i

+

1 E U 0P U n

Together E ^ 2SLS

E 0

=

1 0 X PX n

1

Q + S 22

1

E

1 0 X Pe n

s21 :

(16.14)

In general this is non-zero, except when s21 = 0 (when X is exogenous). It is also close to zero when = 0. Bekker (1994) pointed out that it also has the reverse implication – that when = l=n is large, the bias in the 2SLS estimator will be large. Indeed as ! 1; the expression in (16.14) approaches that in (16.13), indicating that the bias in 2SLS approaches that of OLS as the number of instruments increases. Bekker (1994) showed further that under the alternative asymptotic approximation that is …xed as n ! 1 (so that the number of instruments goes to in…nity proportionately with sample size) then the expression in (16.14) is the probability limit of ^ 2SLS

16.7

Identi…cation Failure

Recall the reduced form equation X2 = Z1

12

+ Z2

22

+ U 2:

The parameter fails to be identi…ed if 22 has de…cient rank. The consequences of identi…cation failure for inference are quite severe. Take the simplest case where k = l = 1 (so there is no Z 1 ): Then the model may be written as yi = xi + ei xi = zi + ui and 22 = = E (zi xi ) =Ezi2 : We see that is identi…ed if and only if 6= 0; which occurs when E (xi zi ) 6= 0. Thus identi…cation hinges on the existence of correlation between the excluded exogenous variable and the included endogenous variable. Suppose this condition fails, so E (xi zi ) = 0: Then by the CLT n

1 X d p zi ei ! N1 n

N 0; E zi2 e2i

(16.15)

i=1

n

n

1 X 1 X d p zi xi = p zi u i ! N 2 n n i=1

therefore

^

N 0; E zi2 u2i

(16.16)

i=1

=

p1 n p1 n

Pn

i=1 zi ei

Pn

i=1 zi xi

d

!

N1 N2

Cauchy;

since the ratio of two normals is Cauchy. This is particularly nasty, as the Cauchy distribution does not have a …nite mean. This result carries over to more general settings, and was examined by Phillips (1989) and Choi and Phillips (1992).

262

CHAPTER 16. ENDOGENEITY

Suppose that identi…cation does not completely fail, but is weak. This occurs when 22 is full rank, but small. This can be handled in an asymptotic analysis by modeling it as local-to-zero, viz 22

1=2

=n

C;

where C is a full rank matrix. The n 1=2 is picked because it provides just the right balancing to allow a rich distribution theory. To see the consequences, once again take the simple case k = l = 1: Here, the instrument xi is weak for zi if = n 1=2 c: Then (16.15) is una¤ected, but (16.16) instead takes the form n

n

n

i=1

i=1

1 X 1 X 1 X 2 p zi xi = p zi + p zi ui n n n i=1

n n 1X 2 1 X p zi u i = zi c + n n i=1

i=1

d

! Qc + N2

therefore ^

d

!

N1 : Qc + N2

As in the case of complete identi…cation failure, we …nd that ^ is inconsistent for and the ^ asymptotic distribution of is non-normal. In addition, standard test statistics have non-standard distributions, meaning that inferences about parameters of interest can be misleading. The distribution theory for this model was developed by Staiger and Stock (1997) and extended to nonlinear GMM estimation by Stock and Wright (2000). Further results on testing were obtained by Wang and Zivot (1998). The bottom line is that it is highly desirable to avoid identi…cation failure. Once again, the equation to focus on is the reduced form X2 = Z1

12

+ Z2

22

+ U2

and identi…cation requires rank( 22 ) = k2 : If k2 = 1; this requires 22 6= 0; which is straightforward to assess using a hypothesis test on the reduced form. Therefore in the case of k2 = 1 (one RHS endogenous variable), one constructive recommendation is to explicitly estimate the reduced form equation for X 2 ; construct the test of 22 = 0, and at a minimum check that the test rejects H0 : 22 = 0. When k2 > 1; 22 6= 0 is not su¢cient for identi…cation. It is not even su¢cient that each column of 22 is non-zero (each column corresponds to a distinct endogenous variable in Z 2 ): So while a minimal check is to test that each columns of 22 is non-zero, this cannot be interpreted as de…nitive proof that 22 has full rank. Unfortunately, tests of de…cient rank are di¢cult to implement. In any event, it appears reasonable to explicitly estimate and report the reduced form equations for Z 2 ; and attempt to assess the likelihood that 22 has de…cient rank.

263

CHAPTER 16. ENDOGENEITY

Exercises 1. Consider the single equation model yi = xi + ei ; where yi and zi are both real-valued (1 1). Let ^ denote the IV estimator of using as an instrument a dummy variable di (takes only the values 0 and 1). Find a simple expression for the IV estimator in this context. 2. In the linear model yi = x0i + ei E (ei j xi ) = 0 suppose 2i = E e2i j xi is known. Show that the GLS estimator of IV estimator using some instrument z i : (Find an expression for z i :)

can be written as an

3. Take the linear model y = X + e: be ^ and the OLS residual be e ^ = y X ^. Let the IV estimator for using some instrument Z be ~ and the IV residual be e ~ = y X ~. If X is indeed endogeneous, will IV “…t” better than OLS, in the sense that e ~0 e ~ 0; E (yt k et ) = 0:

268

CHAPTER 17. UNIVARIATE TIME SERIES

17.3

Stationarity of AR(1) Process

A mean-zero AR(1) is yt = y t Assume that et is iid, E(et ) = 0 and By back-substitution, we …nd

Ee2t

2

=

+ et :

1

< 1:

yt = et + et 1 X k et =

2

+

1

et

2

+ :::

k:

k=0

ke t k

Loosely speaking, this series converges if the sequence when j j < 1:

gets small as k ! 1: This occurs

Theorem 17.3.1 If and only if j j < 1 then yt is strictly stationary and ergodic.

We can compute the moments of yt using the in…nite sum: Eyt =

1 X

k

E (et

k)

=0

var (et

k)

=

k=0

var(yt ) =

1 X

2

2k

k=0

1

2

:

If the equation for yt has an intercept, the above results are unchanged, except that the mean of yt can be computed from the relationship Eyt = and solving for Eyt = Eyt

17.4

1

+ Eyt

we …nd Eyt = =(1

1;

):

Lag Operator

An algebraic construct which is useful for the analysis of autoregressive models is the lag operator. De…nition 17.4.1 The lag operator L satis…es Lyt = yt

De…ning L2 = LL; we see that L2 yt = Lyt 1 = yt The AR(1) model can be written in the format

2:

In general, Lk yt = yt

yt

yt

(1

L) yt = et :

1

1:

k:

= et

or The operator (L) = (1 L) is a polynomial in the operator L: We say that the root of the polynomial is 1= ; since (z) = 0 when z = 1= : We call (L) the autoregressive polynomial of yt . From Theorem 17.3.1, an AR(1) is stationary i¤ j j < 1: Note that an equivalent way to say this is that an AR(1) is stationary i¤ the root of the autoregressive polynomial is larger than one (in absolute value).

269

CHAPTER 17. UNIVARIATE TIME SERIES

17.5

Stationarity of AR(k)

The AR(k) model is yt =

1 yt 1

yt

1 Lyt

+

2 yt 2

+

+

k yt k

+ et :

k k L yt

= et ;

Using the lag operator, 2 2 L yt

or (L)yt = et where (L) = 1

2 2L

1L

kL

k

:

We call (L) the autoregressive polynomial of yt : The Fundamental Theorem of Algebra says that any polynomial can be factored as (z) = 1

1

1

z

1

2

1

z

1

1

k

z

where the 1 ; :::; k are the complex roots of (z); which satisfy ( j ) = 0: We know that an AR(1) is stationary i¤ the absolute value of the root of its autoregressive polynomial is larger than one. For an AR(k), the requirement is that all roots are larger than one. Let j j denote the modulus of a complex number : Theorem 17.5.1 The AR(k) is strictly stationary and ergodic if and only if j j j > 1 for all j:

One way of stating this is that “All roots lie outside the unit circle.” If one of the roots equals 1, we say that (L); and hence yt ; “has a unit root”. This is a special case of non-stationarity, and is of great interest in applied time series.

17.6

Estimation

Let 1 yt

xt = =

1

yt

1

yt

2 k

2

0

k

0

:

Then the model can be written as yt = x0t + et : The OLS estimator is

^ = X 0X

1

X 0 y:

To study ^ ; it is helpful to de…ne the process ut = xt et : Note that ut is a MDS, since E (ut j Ft

1)

= E (xt et j Ft

1)

= xt E (et j Ft

1)

= 0:

By Theorem 17.1.1, it is also strictly stationary and ergodic. Thus T T 1X 1X p xt et = ut ! E (ut ) = 0: T T t=1

t=1

(17.1)

270

CHAPTER 17. UNIVARIATE TIME SERIES

The vector xt is strictly stationary and ergodic, and by Theorem 17.1.1, so is xt x0t : Thus by the Ergodic Theorem, T 1X p xt x0t ! E xt x0t = Q: T t=1

Combined with (17.1) and the continuous mapping theorem, we see that ^=

+

T 1X xt x0t T t=1

!

1

T 1X xt et T t=1

!

p

!Q

1

0 = 0:

We have shown the following:

Theorem 17.6.1 If the AR(k) process yt is strictly stationary and ergodic p and Eyt2 < 1; then ^ ! as T ! 1:

17.7

Asymptotic Distribution Theorem 17.7.1 MDS CLT. If ut is a strictly stationary and ergodic MDS and E (ut u0t ) = < 1; then as T ! 1; T 1 X d p ut ! N (0; ) : T t=1

Since xt et is a MDS, we can apply Theorem 17.7.1 to see that T 1 X d p xt et ! N (0; ) ; T t=1

where = E(xt x0t e2t ):

Theorem 17.7.2 If the AR(k) process yt is strictly stationary and ergodic and Eyt4 < 1; then as T ! 1; p

T ^

d

! N 0; Q

1

Q

1

:

This is identical in form to the asymptotic distribution of OLS in cross-section regression. The implication is that asymptotic inference is the same. In particular, the asymptotic covariance matrix is estimated just as in the cross-section case.

271

CHAPTER 17. UNIVARIATE TIME SERIES

17.8

Bootstrap for Autoregressions

In the non-parametric bootstrap, we constructed the bootstrap sample by randomly resampling from the data values fyt ; xt g: This creates an iid bootstrap sample. Clearly, this cannot work in a time-series application, as this imposes inappropriate independence. Brie‡y, there are two popular methods to implement bootstrap resampling for time-series data. Method 1: Model-Based (Parametric) Bootstrap. 1. Estimate ^ and residuals e^t : 2. Fix an initial condition (y

k+1 ; y k+2 ; :::; y0 ):

3. Simulate iid draws ei from the empirical distribution of the residuals f^ e1 ; :::; e^T g: 4. Create the bootstrap series yt by the recursive formula yt = ^ + ^ 1 yt

1

+ ^ 2 yt

2

+

+ ^ k yt

k

+ et :

This construction imposes homoskedasticity on the errors ei ; which may be di¤erent than the properties of the actual ei : It also presumes that the AR(k) structure is the truth. Method 2: Block Resampling 1. Divide the sample into T =m blocks of length m: 2. Resample complete blocks. For each simulated sample, draw T =m blocks. 3. Paste the blocks together to create the bootstrap time-series yt : 4. This allows for arbitrary stationary serial correlation, heteroskedasticity, and for modelmisspeci…cation. 5. The results may be sensitive to the block length, and the way that the data are partitioned into blocks. 6. May not work well in small samples.

17.9

Trend Stationarity

yt =

0

+

1t

St =

1 St 1

+ St

+

(17.2)

2 St 2

+

+

k St l

+ et ;

(17.3)

or yt =

0

+

1t

+

1 yt 1

+

2 yt 1

+

+

k yt k

+ et :

(17.4)

There are two essentially equivalent ways to estimate the autoregressive parameters ( 1 ; :::;

k ):

You can estimate (17.4) by OLS. You can estimate (17.2)-(17.3) sequentially by OLS. That is, …rst estimate (17.2), get the residual S^t ; and then perform regression (17.3) replacing St with S^t : This procedure is sometimes called Detrending.

272

CHAPTER 17. UNIVARIATE TIME SERIES

The reason why these two procedures are (essentially) the same is the Frisch-Waugh-Lovell theorem. Seasonal E¤ects There are three popular methods to deal with seasonal data. Include dummy variables for each season. This presumes that “seasonality” does not change over the sample. Use “seasonally adjusted” data. The seasonal factor is typically estimated by a two-sided weighted average of the data for that season in neighboring years. Thus the seasonally adjusted data is a “…ltered” series. This is a ‡exible approach which can extract a wide range of seasonal factors. The seasonal adjustment, however, also alters the time-series correlations of the data. First apply a seasonal di¤erencing operator. If s is the number of seasons (typically s = 4 or s = 12); yt s ; s yt = yt or the season-to-season change. The series s yt is clearly free of seasonality. But the long-run trend is also eliminated, and perhaps this was of relevance.

17.10

Testing for Omitted Serial Correlation

For simplicity, let the null hypothesis be an AR(1): yt =

+ yt

1

+ ut :

(17.5)

We are interested in the question if the error ut is serially correlated. We model this as an AR(1): ut = ut

1

+ et

(17.6)

with et a MDS. The hypothesis of no omitted serial correlation is H0 :

=0

H1 :

6= 0:

We want to test H0 against H1 : To combine (17.5) and (17.6), we take (17.5) and lag the equation once: yt We then multiply this by yt

1

=

+ yt

2

+ ut

1:

and subtract from (17.5), to …nd yt

1

=

+ yt

1

yt

1

+ ut

ut

1;

or yt = (1

) + ( + ) yt

1

yt

2

+ et = AR(2):

Thus under H0 ; yt is an AR(1), and under H1 it is an AR(2). H0 may be expressed as the restriction that the coe¢cient on yt 2 is zero. An appropriate test of H0 against H1 is therefore a Wald test that the coe¢cient on yt 2 is zero. (A simple exclusion test). In general, if the null hypothesis is that yt is an AR(k), and the alternative is that the error is an AR(m), this is the same as saying that under the alternative yt is an AR(k+m), and this is equivalent to the restriction that the coe¢cients on yt k 1 ; :::; yt k m are jointly zero. An appropriate test is the Wald test of this restriction.

273

CHAPTER 17. UNIVARIATE TIME SERIES

17.11

Model Selection

What is the appropriate choice of k in practice? This is a problem of model selection. One approach to model selection is to choose k based on a Wald tests. Another is to minimize the AIC or BIC information criterion, e.g. AIC(k) = log ^ 2 (k) +

2k ; T

where ^ 2 (k) is the estimated residual variance from an AR(k) One ambiguity in de…ning the AIC criterion is that the sample available for estimation changes as k changes. (If you increase k; you need more initial conditions.) This can induce strange behavior in the AIC. The best remedy is to …x a upper value k; and then reserve the …rst k as initial conditions, and then estimate the models AR(1), AR(2), ..., AR(k) on this (uni…ed) sample.

17.12

Autoregressive Unit Roots

The AR(k) model is (L)yt =

+ et

(L) = 1

k kL :

1L

As we discussed before, yt has a unit root when (1) = 0; or 1

+

2

+

+

k

= 1:

In this case, yt is non-stationary. The ergodic theorem and MDS CLT do not apply, and test statistics are asymptotically non-normal. A helpful way to write the equation is the so-called Dickey-Fuller reparameterization: yt =

+

0 yt 1

+

yt

1

1

+

+

k 1

yt

(k 1)

+ et :

(17.7)

These models are equivalent linear transformations of one another. The DF parameterization is convenient because the parameter 0 summarizes the information about the unit root, since (1) = 0 : To see this, observe that the lag polynomial for the yt computed from (17.7) is (1

L)

0L

1 (L

L2 )

k 1 (L

k 1

Lk )

But this must equal (L); as the models are equivalent. Thus (1) = (1

1)

(1

0

1)

(1

1) =

0:

Hence, the hypothesis of a unit root in yt can be stated as H0 : Note that the model is stationary if

0

0

= 0:

< 0: So the natural alternative is H1 :

0

< 0:

Under H0 ; the model for yt is yt =

+

1

yt

1

+

+

k 1

yt

(k 1)

+ et ;

which is an AR(k-1) in the …rst-di¤erence yt : Thus if yt has a (single) unit root, then yt is a stationary AR process. Because of this property, we say that if yt is non-stationary but d yt is stationary, then yt is “integrated of order d”; or I(d): Thus a time series with unit root is I(1):

274

CHAPTER 17. UNIVARIATE TIME SERIES

Since 0 is the parameter of a linear regression, the natural test statistic is the t-statistic for H0 from OLS estimation of (17.7). Indeed, this is the most popular unit root test, and is called the Augmented Dickey-Fuller (ADF) test for a unit root. It would seem natural to assess the signi…cance of the ADF statistic using the normal table. However, under H0 ; yt is non-stationary, so conventional normal asymptotics are invalid. An alternative asymptotic framework has been developed to deal with non-stationary data. We do not have the time to develop this theory in detail, but simply assert the main results.

Theorem 17.12.1 Dickey-Fuller Theorem. Assume 0 = 0: As T ! 1; d

T ^ 0 ! (1

1

ADF =

k 1 ) DF

2

^0 ! DFt : s(^ 0 )

The limit distributions DF and DFt are non-normal. They are skewed to the left, and have negative means. The …rst result states that ^ 0 converges to its true value (of zero) at rate T; rather than the conventional rate of T 1=2 : This is called a “super-consistent” rate of convergence. The second result states that the t-statistic for ^ 0 converges to a limit distribution which is non-normal, but does not depend on the parameters : This distribution has been extensively tabulated, and may be used for testing the hypothesis H0 : Note: The standard error s(^ 0 ) is the conventional (“homoskedastic”) standard error. But the theorem does not require an assumption of homoskedasticity. Thus the Dickey-Fuller test is robust to heteroskedasticity. Since the alternative hypothesis is one-sided, the ADF test rejects H0 in favor of H1 when ADF < c; where c is the critical value from the ADF table. If the test rejects H0 ; this means that the evidence points to yt being stationary. If the test does not reject H0 ; a common conclusion is that the data suggests that yt is non-stationary. This is not really a correct conclusion, however. All we can say is that there is insu¢cient evidence to conclude whether the data are stationary or not. We have described the test for the setting of with an intercept. Another popular setting includes as well a linear time trend. This model is yt =

1

+

2t

+

0 yt 1

+

1

yt

1

+

+

k 1

yt

(k 1)

+ et :

(17.8)

This is natural when the alternative hypothesis is that the series is stationary about a linear time trend. If the series has a linear trend (e.g. GDP, Stock Prices), then the series itself is nonstationary, but it may be stationary around the linear time trend. In this context, it is a silly waste of time to …t an AR model to the level of the series without a time trend, as the AR model cannot conceivably describe this data. The natural solution is to include a time trend in the …tted OLS equation. When conducting the ADF test, this means that it is computed as the t-ratio for 0 from OLS estimation of (17.8). If a time trend is included, the test procedure is the same, but di¤erent critical values are required. The ADF test has a di¤erent distribution when the time trend has been included, and a di¤erent table should be consulted. Most texts include as well the critical values for the extreme polar case where the intercept has been omitted from the model. These are included for completeness (from a pedagogical perspective) but have no relevance for empirical practice where intercepts are always included.

Chapter 18

Multivariate Time Series A multivariate time series y t is a vector process m 1. Let Ft 1 = (y t 1 ; y t 2 ; :::) be all lagged information at time t: The typical goal is to …nd the conditional expectation E (y t j Ft 1 ) : Note that since y t is a vector, this conditional expectation is also a vector.

18.1

Vector Autoregressions (VARs)

A VAR model speci…es that the conditional mean is a function of only a …nite number of lags: E (y t j Ft

1)

= E yt j yt

1 ; :::; y t k

:

A linear VAR speci…es that this conditional mean is linear in the arguments: E yt j yt

1 ; :::; y t k

= a0 + A1 y t

1

+ A2 y t

Observe that a0 is m 1,and each of A1 through Ak are m De…ning the m 1 regression error et = y t

E (y t j Ft

2

+

Ak y t

k:

m matrices.

1) ;

we have the VAR model y t = a0 + A1 y t E (et j Ft

1)

1

+ A2 y t

2

+

Ak y t

k

+ et

= 0:

Alternatively, de…ning the mk + 1 vector 0

and the m

1

B yt B B xt = B y t B .. @ . yt

(mk + 1) matrix A=

1 2

k

1 C C C C C A

a0 A1 A2

Ak

;

then y t = Axt + et : The VAR model is a system of m equations. One way to write this is to let a0j be the jth row of A. Then the VAR system can be written as the equations Yjt = a0j xt + ejt : Unrestricted VARs were introduced to econometrics by Sims (1980). 275

276

CHAPTER 18. MULTIVARIATE TIME SERIES

18.2

Estimation

Consider the moment conditions E (xt ejt ) = 0; j = 1; :::; m: These are implied by the VAR model, either as a regression, or as a linear projection. The GMM estimator corresponding to these moment conditions is equation-by-equation OLS a ^j = (X 0 X)

1

X 0yj :

An alternative way to compute this is as follows. Note that a ^0j = y 0j X(X 0 X)

1

:

^ we …nd And if we stack these to create the estimate A; 0 1 y 01 B y0 C 2 B C ^ A = B . C X(X 0 X) @ .. A

1

y 0m+1

= Y 0 X(X 0 X)

1

;

where Y =

y1 y2

ym

the T m matrix of the stacked y 0t : This (system) estimator is known as the SUR (Seemingly Unrelated Regressions) estimator, and was originally derived by Zellner (1962)

18.3

Restricted VARs

The unrestricted VAR is a system of m equations, each with the same set of regressors. A restricted VAR imposes restrictions on the system. For example, some regressors may be excluded from some of the equations. Restrictions may be imposed on individual equations, or across equations. The GMM framework gives a convenient method to impose such restrictions on estimation.

18.4

Single Equation from a VAR

Often, we are only interested in a single equation out of a VAR system. This takes the form yjt = a0j xt + et ; and xt consists of lagged values of yjt and the other ylt0 s: In this case, it is convenient to re-de…ne the variables. Let yt = yjt ; and z t be the other variables. Let et = ejt and = aj : Then the single equation takes the form yt = x0t + et ; (18.1) and xt =

h

1 yt

1

yt

k

z 0t

1

This is just a conventional regression with time series data.

z 0t

k

0

i

:

277

CHAPTER 18. MULTIVARIATE TIME SERIES

18.5

Testing for Omitted Serial Correlation

Consider the problem of testing for omitted serial correlation in equation (18.1). Suppose that et is an AR(1). Then yt = x0t + et et = E (ut j Ft

1)

et

1

+ ut

(18.2)

= 0:

Then the null and alternative are H0 :

=0

H1 :

6= 0:

Take the equation yt = x0t + et ; and subtract o¤ the equation once lagged multiplied by ; to get yt

yt

1

= =

x0t + et x0t

xt

x0t 1

1

+ et

+ et et

1

1;

or yt = y t

1

+ x0t + x0t

1

+ ut ;

(18.3)

which is a valid regression model. So testing H0 versus H1 is equivalent to testing for the signi…cance of adding (yt 1 ; xt 1 ) to the regression. This can be done by a Wald test. We see that an appropriate, general, and simple way to test for omitted serial correlation is to test the signi…cance of extra lagged values of the dependent variable and regressors. You may have heard of the Durbin-Watson test for omitted serial correlation, which once was very popular, and is still routinely reported by conventional regression packages. The DW test is appropriate only when regression yt = x0t + et is not dynamic (has no lagged values on the RHS), and et is iid N(0; 2 ): Otherwise it is invalid. Another interesting fact is that (18.2) is a special case of (18.3), under the restriction = : This restriction, which is called a common factor restriction, may be tested if desired. If valid, the model (18.2) may be estimated by iterated GLS. (A simple version of this estimator is called Cochrane-Orcutt.) Since the common factor restriction appears arbitrary, and is typically rejected empirically, direct estimation of (18.2) is uncommon in recent applications.

18.6

Selection of Lag Length in an VAR

If you want a data-dependent rule to pick the lag length k in a VAR, you may either use a testingbased approach (using, for example, the Wald statistic), or an information criterion approach. The formula for the AIC and BIC are p AIC(k) = log det ^ (k) + 2 T p log(T ) BIC(k) = log det ^ (k) + T T X ^ (k) = 1 e ^t (k)^ et (k)0 T t=1

p = m(km + 1)

where p is the number of parameters in the model, and e ^t (k) is the OLS residual vector from the model with k lags. The log determinant is the criterion from the multivariate normal likelihood.

278

CHAPTER 18. MULTIVARIATE TIME SERIES

18.7

Granger Causality

Partition the data vector into (y t ; z t ): De…ne the two information sets F1t =

yt; yt

F2t =

1 ; y t 2 ; :::

yt; zt; yt

1 ; z t 1 ; y t 2 ; z t 2 ; ; :::

The information set F1t is generated only by the history of y t ; and the information set F2t is generated by both y t and z t : The latter has more information. We say that z t does not Granger-cause y t if E (y t j F1;t

1)

= E (y t j F2;t

1) :

That is, conditional on information in lagged y t ; lagged z t does not help to forecast y t : If this condition does not hold, then we say that z t Granger-causes y t : The reason why we call this “Granger Causality” rather than “causality” is because this is not a physical or structure de…nition of causality. If z t is some sort of forecast of the future, such as a futures price, then z t may help to forecast y t even though it does not “cause” y t : This de…nition of causality was developed by Granger (1969) and Sims (1972). In a linear VAR, the equation for y t is yt =

+

1yt 1

+

+

k yt k

+ z 0t

1 1

+

+ z 0t

k k

+ et :

In this equation, z t does not Granger-cause y t if and only if H0 :

1

=

2

=

=

k

= 0:

This may be tested using an exclusion (Wald) test. This idea can be applied to blocks of variables. That is, y t and/or z t can be vectors. The hypothesis can be tested by using the appropriate multivariate Wald test. If it is found that z t does not Granger-cause y t ; then we deduce that our time-series model of E (y t j Ft 1 ) does not require the use of z t : Note, however, that z t may still be useful to explain other features of y t ; such as the conditional variance.

Clive W. J. Granger Clive Granger (1934-2009) of England was one of the leading …gures in time-series econometrics, and co-winner in 2003 of the Nobel Memorial Prize in Economic Sciences (along with Robert Engle). In addition to formalizing the de…nition of causality known as Granger causality, he invented the concept of cointegration, introduced spectral methods into econometrics, and formalized methods for the combination of forecasts.

18.8

Cointegration

The idea of cointegration is due to Granger (1981), and was articulated in detail by Engle and Granger (1987).

De…nition 18.8.1 The m 1 series y t is cointegrated if y t is I(1) yet there exists ; m r, of rank r; such that z t = 0 y t is I(0): The r vectors in are called the cointegrating vectors.

279

CHAPTER 18. MULTIVARIATE TIME SERIES

If the series y t is not cointegrated, then r = 0: If r = m; then y t is I(0): For 0 < r < m; y t is I(1) and cointegrated. In some cases, it may be believed that is known a priori. Often, = (1 1)0 : For example, 0 if y t is a pair of interest rates, then = (1 1) speci…es that the spread (the di¤erence in returns) is stationary. If y = (log(Consumption) log(Income))0 ; then = (1 1)0 speci…es that log(Consumption=Income) is stationary. In other cases, may not be known. If y t is cointegrated with a single cointegrating vector (r = 1); then it turns out that can be consistently estimated by an OLS regression of one component of y t on the others. Thus y t = p (Y1t ; Y2t ) and = ( 1 2 ) and normalize 1 = 1: Then ^ 2 = (y 02 y 2 ) 1 y 02 y 1 ! 2 : Furthermore d this estimation is super-consistent: T ( ^ ) ! Limit; as …rst shown by Stock (1987). This 2

2

is not, in general, a good method to estimate ; but it is useful in the construction of alternative estimators and tests. We are often interested in testing the hypothesis of no cointegration: H0 : r = 0 H1 : r > 0:

Suppose that is known, so z t = 0 y t is known. Then under H0 z t is I(1); yet under H1 z t is I(0): Thus H0 can be tested using a univariate ADF test on z t : When is unknown, Engle and Granger (1987) suggested using an ADF test on the estimated 0 residual z^t = ^ y t ; from OLS of y1t on y2t : Their justi…cation was Stock’s result that ^ is superconsistent under H1 : Under H0 ; however, ^ is not consistent, so the ADF critical values are not appropriate. The asymptotic distribution was worked out by Phillips and Ouliaris (1990). When the data have time trends, it may be necessary to include a time trend in the estimated cointegrating regression. Whether or not the time trend is included, the asymptotic distribution of the test is a¤ected by the presence of the time trend. The asymptotic distribution was worked out in B. Hansen (1992).

18.9

Cointegrated VARs

We can write a VAR as A(L)y t = et A(L) = I

A1 L

Ak Lk

A2 L2

or alternatively as yt =

yt

1

+ D(L) y t

1

+ et

where =

A(1)

=

I + A1 + A2 +

+ Ak :

Theorem 18.9.1 Granger Representation Theorem y t is cointegrated with m r if and only if rank( ) = r and where is m r, rank ( ) = r:

=

0

280

CHAPTER 18. MULTIVARIATE TIME SERIES

Thus cointegration imposes a restriction upon the parameters of a VAR. The restricted model can be written as yt =

0

yt

yt =

zt

1

1

+ D(L) y t

+ D(L) y t

1

1

+ et

+ et :

If is known, this can be estimated by OLS of y t on z t 1 and the lags of y t : If is unknown, then estimation is done by “reduced rank regression”, which is least-squares subject to the stated restriction. Equivalently, this is the MLE of the restricted parameters under the assumption that et is iid N(0; ): One di¢culty is that is not identi…ed without normalization. When r = 1; we typically just normalize one element to equal unity. When r > 1; this does not work, and di¤erent authors have adopted di¤erent identi…cation schemes. In the context of a cointegrated VAR estimated by reduced rank regression, it is simple to test for cointegration by testing the rank of : These tests are constructed as likelihood ratio (LR) tests. As they were discovered by Johansen (1988, 1991, 1995), they are typically called the “Johansen Max and Trace” tests. Their asymptotic distributions are non-standard, and are similar to the Dickey-Fuller distributions.

Chapter 19

Limited Dependent Variables A “limited dependent variable” y is one which takes a “limited” set of values. The most common cases are Binary: y 2 f0; 1g Multinomial: y 2 f0; 1; 2; :::; kg Integer: y 2 f0; 1; 2; :::g Censored: y 2 R+ The traditional approach to the estimation of limited dependent variable (LDV) models is parametric maximum likelihood. A parametric model is constructed, allowing the construction of the likelihood function. A more modern approach is semi-parametric, eliminating the dependence on a parametric distributional assumption. We will discuss only the …rst (parametric) approach, due to time constraints. They still constitute the majority of LDV applications. If, however, you were to write a thesis involving LDV estimation, you would be advised to consider employing a semi-parametric estimation approach. For the parametric approach, estimation is by MLE. A major practical issue is construction of the likelihood function.

19.1

Binary Choice

The dependent variable yi 2 f0; 1g: This represents a Yes/No outcome. Given some regressors xi ; the goal is to describe Pr (yi = 1 j xi ) ; as this is the full conditional distribution. The linear probability model speci…es that Pr (yi = 1 j xi ) = x0i : As Pr (yi = 1 j xi ) = E (yi j xi ) ; this yields the regression: yi = x0i + ei which can be estimated by OLS. However, the linear probability model does not impose the restriction that 0 Pr (yi j xi ) 1: Even so estimation of a linear probability model is a useful starting point for subsequent analysis. The standard alternative is to use a function of the form Pr (yi = 1 j xi ) = F x0i where F ( ) is a known CDF, typically assumed to be symmetric about zero, so that F (u) = 1 F ( u): The two standard choices for F are Logistic: F (u) = (1 + e

u) 1 :

281

282

CHAPTER 19. LIMITED DEPENDENT VARIABLES Normal: F (u) =

(u):

If F is logistic, we call this the logit model, and if F is normal, we call this the probit model. This model is identical to the latent variable model yi

= x0i + ei

ei

F() 1 = 0

yi

if yi > 0 : otherwise

For then Pr (yi = 1 j xi ) = Pr (yi > 0 j xi )

= Pr x0i + ei > 0 j xi x0i

= Pr ei >

= 1

j xi

x0i

F

= F x0i

:

Estimation is by maximum likelihood. To construct the likelihood, we need the conditional distribution of an individual observation. Recall that if y is Bernoulli, such that Pr(y = 1) = p and Pr(y = 0) = 1 p, then we can write the density of y as f (y) = py (1

y

p)1

;

y = 0; 1:

In the Binary choice model, yi is conditionally Bernoulli with Pr (yi = 1 j xi ) = pi = F (x0i ) : Thus the conditional density is f (yi j xi ) = pyi i (1

pi )1 yi

= F x0i

yi

F x0i

(1

)1

yi

:

Hence the log-likelihood function is log L( ) = =

n X

i=1 n X

log f (yi j xi ) log F x0i

yi

(1

F x0i

)1

yi

i=1

=

n X

yi log F x0i

+ (1

i=1

=

X

log F x0i

+

X

yi ) log(1

log(1

F x0i

F x0i

)

):

yi =0

yi =1

The MLE ^ is the value of which maximizes log L( ): Standard errors and test statistics are computed by asymptotic approximations. Details of such calculations are left to more advanced courses.

19.2

Count Data

If y 2 f0; 1; 2; :::g; a typical approach is to employ Poisson regression. This model speci…es that Pr (yi = k j xi ) = i

exp (

i)

k! = exp(x0i ):

k i

;

k = 0; 1; 2; :::

283

CHAPTER 19. LIMITED DEPENDENT VARIABLES The conditional density is the Poisson with parameter picked to ensure that i > 0. The log-likelihood function is log L( ) =

n X i=1

log f (yi j xi ) =

n X

i:

The functional form for

exp(x0i ) + yi x0i

i

has been

log(yi !) :

i=1

The MLE is the value ^ which maximizes log L( ): Since E (yi j xi ) = i = exp(x0i ) is the conditional mean, this motivates the label Poisson “regression.” Also observe that the model implies that var (yi j xi ) =

i

= exp(x0i );

so the model imposes the restriction that the conditional mean and variance of yi are the same. This may be considered restrictive. A generalization is the negative binomial.

19.3

Censored Data

The idea of “censoring” is that some data above or below a threshold are mis-reported at the threshold. Thus the model is that there is some latent process yi with unbounded support, but we observe only 0 if yi yi : (19.1) yi = 0 if yi < 0 (This is written for the case of the threshold being zero, any known value can substitute.) The observed data yi therefore come from a mixed continuous/discrete distribution. Censored models are typically applied when the data set has a meaningful proportion (say 5% or higher) of data at the boundary of the sample support. The censoring process may be explicit in data collection, or it may be a by-product of economic constraints. An example of a data collection censoring is top-coding of income. In surveys, incomes above a threshold are typically reported at the threshold. The …rst censored regression model was developed by Tobin (1958) to explain consumption of durable goods. Tobin observed that for many households, the consumption level (purchases) in a particular period was zero. He proposed the latent variable model yi

= x0i + ei

ei

iid N(0;

2

)

with the observed variable yi generated by the censoring equation (19.1). This model (now called the Tobit) speci…es that the latent (or ideal) value of consumption may be negative (the household would prefer to sell than buy). All that is reported is that the household purchased zero units of the good. The naive approach to estimate is to regress yi on xi . This does not work because regression estimates E (yi j xi ) ; not E (yi j xi ) = x0i ; and the latter is of interest. Thus OLS will be biased for the parameter of interest : [Note: it is still possible to estimate E (yi j xi ) by LS techniques. The Tobit framework postulates that this is not inherently interesting, that the parameter of is de…ned by an alternative statistical structure.]

284

CHAPTER 19. LIMITED DEPENDENT VARIABLES

Consistent estimation will be achieved by the MLE. To construct the likelihood, observe that the probability of being censored is Pr (yi = 0 j xi ) = Pr (yi < 0 j xi )

= Pr x0i + ei < 0 j xi x0i ei < j xi = Pr x0i

=

:

The conditional density function above zero is normal: y

1

Therefore, the density function for y

x0i

;

y > 0:

0 can be written as x0i

f (y j xi ) =

1(y=0)

z

1

x0i

1(y>0)

;

where 1 ( ) is the indicator function. Hence the log-likelihood is a mixture of the probit and the normal: log L( ) =

n X i=1

=

X

log f (yi j xi ) log

x0i

+

X

log

1

yi

x0i

:

yi >0

yi =0

The MLE is the value ^ which maximizes log L( ):

19.4

Sample Selection

The problem of sample selection arises when the sample is a non-random selection of potential observations. This occurs when the observed data is systematically di¤erent from the population of interest. For example, if you ask for volunteers for an experiment, and they wish to extrapolate the e¤ects of the experiment on a general population, you should worry that the people who volunteer may be systematically di¤erent from the general population. This has great relevance for the evaluation of anti-poverty and job-training programs, where the goal is to assess the e¤ect of “training” on the general population, not just on the volunteers. A simple sample selection model can be written as the latent model yi = x0i + e1i Ti = 1 z 0i + e0i > 0 where 1 ( ) is the indicator function. The dependent variable yi is observed if (and only if) Ti = 1: Else it is unobserved. For example, yi could be a wage, which can be observed only if a person is employed. The equation for Ti is an equation specifying the probability that the person is employed. The model is often completed by specifying that the errors are jointly normal e0i e1i

N 0;

1 2

:

285

CHAPTER 19. LIMITED DEPENDENT VARIABLES It is presumed that we observe fxi ; z i ; Ti g for all observations. Under the normality assumption, e1i = e0i + vi ; where vi is independent of e0i that

N(0; 1): A useful fact about the standard normal distribution is E (e0i j e0i >

x) = (x) =

(x) ; (x)

and the function (x) is called the inverse Mills ratio. The naive estimator of is OLS regression of yi on xi for those observations for which yi is available. The problem is that this is equivalent to conditioning on the event fTi = 1g: However, E (e1i j Ti = 1; z i ) = E e1i j fe0i > =

=

E e0i j fe0i > z 0i

z 0i g; z i

;

z 0i g; z i + E vi j fe0i >

z 0i g; z i

which is non-zero. Thus e1i =

z 0i

+ ui ;

where E (ui j Ti = 1; z i ) = 0: Hence yi = x0i +

z 0i

+ ui

(19.2)

is a valid regression equation for the observations for which Ti = 1: Heckman (1979) observed that we could consistently estimate and from this equation, if were known. It is unknown, but also can be consistently estimated by a Probit model for selection. The “Heckit” estimator is thus calculated as follows Estimate ^ from a Probit, using regressors z i : The binary dependent variable is Ti : Estimate ^ ; ^ from OLS of yi on xi and (z 0i ^ ): The OLS standard errors will be incorrect, as this is a two-step estimator. They can be corrected using a more complicated formula. Or, alternatively, by viewing the Probit/OLS estimation equations as a large joint GMM problem. The Heckit estimator is frequently used to deal with problems of sample selection. However, the estimator is built on the assumption of normality, and the estimator can be quite sensitive to this assumption. Some modern econometric research is exploring how to relax the normality assumption. The estimator can also work quite poorly if (z 0i ^ ) does not have much in-sample variation. This can happen if the Probit equation does not “explain” much about the selection choice. Another potential problem is that if z i = xi ; then (z 0i ^ ) can be highly collinear with xi ; so the second step OLS estimator will not be able to precisely estimate : Based this observation, it is typically recommended to …nd a valid exclusion restriction: a variable should be in z i which is not in xi : If this is valid, it will ensure that (z 0i ^ ) is not collinear with xi ; and hence improve the second stage estimator’s precision.

Chapter 20

Panel Data A panel is a set of observations on individuals, collected over time. An observation is the pair fyit ; xit g; where the i subscript denotes the individual, and the t subscript denotes time. A panel may be balanced : fyit ; xit g : t = 1; :::; T ; i = 1; :::; n; or unbalanced : fyit ; xit g : For i = 1; :::; n;

20.1

t = ti ; :::; ti :

Individual-E¤ects Model

The standard panel data speci…cation is that there is an individual-speci…c e¤ect which enters linearly in the regression yit = x0it + ui + eit : The typical maintained assumptions are that the individuals i are mutually independent, that ui and eit are independent, that eit is iid across individuals and time, and that eit is uncorrelated with xit : OLS of yit on xit is called pooled estimation. It is consistent if E (xit ui ) = 0

(20.1)

If this condition fails, then OLS is inconsistent. (20.1) fails if the individual-speci…c unobserved e¤ect ui is correlated with the observed explanatory variables xit : This is often believed to be plausible if ui is an omitted variable. If (20.1) is true, however, OLS can be improved upon via a GLS technique. In either event, OLS appears a poor estimation choice. Condition (20.1) is called the random e¤ects hypothesis. It is a strong assumption, and most applied researchers try to avoid its use.

20.2

Fixed E¤ects

This is the most common technique for estimation of non-dynamic linear panel regressions. The motivation is to allow ui to be arbitrary, and have arbitrary correlated with xi : The goal is to eliminate ui from the estimator, and thus achieve invariance. There are several derivations of the estimator. First, let 8 if i = j < 1 ; dij = : 0 else 286

287

CHAPTER 20. PANEL DATA and

1 di1 C B di = @ ... A ; din 0

1 dummy vector with a “1” in the i0 th place. 0 u1 B .. u=@ .

an n

un

Then note that

Let 1 C A:

ui = d0i u; and yit = x0it + d0i u + eit :

(20.2)

Observe that E (eit j xit ; di ) = 0; so (20.2) is a valid regression, with di as a regressor along with xi : ^ : Conventional inference applies. OLS on (20.2) yields estimator ^ ; u Observe that This is generally consistent. If xit contains an intercept, it will be collinear with di ; so the intercept is typically omitted from xit : Any regressor in xit which is constant over time for all individuals (e.g., their gender) will be collinear with di ; so will have to be omitted. There are n + k regression parameters, which is quite large as typically n is very large. Computationally, you do not want to actually implement conventional OLS estimation, as the parameter space is too large. OLS estimation of proceeds by the FWL theorem. Stacking the observations together: y = X + Du + e; then by the FWL theorem, ^ = =

X 0 (I

P D) X 1

X 0X

X 0y

1

X 0 (I

P D) y

;

where y

= y

X

= X

D(D 0 D) D(D 0 D)

1

D0 y 1

D 0 X:

Since the regression of yit on di is a regression onto individual-speci…c dummies, the predicted value from these regressions is the individual speci…c mean y i ; and the residual is the demean value yit = yit

yi:

The …xed e¤ects estimator ^ is OLS of yit on xit , the dependent variable and regressors in deviationfrom-mean form.

288

CHAPTER 20. PANEL DATA Another derivation of the estimator is to take the equation yit = x0it + ui + eit ; and then take individual-speci…c means by taking the average for the i0 th individual: ti ti ti 1 X 1 X 1 X yit = x0it + ui + eit Ti t=t Ti t=t Ti t=t i

i

i

or y i = x0i + ui + ei : Subtracting, we …nd yit = xit0 + eit ; which is free of the individual-e¤ect ui :

20.3

Dynamic Panel Regression

A dynamic panel regression has a lagged dependent variable yit = yit

1

+ x0it + ui + eit :

(20.3)

This is a model suitable for studying dynamic behavior of individual agents. Unfortunately, the …xed e¤ects estimator is inconsistent, at least if T is held …nite as n ! 1: This is because the sample mean of yit 1 is correlated with that of eit : The standard approach to estimate a dynamic panel is to combine …rst-di¤erencing with IV or GMM. Taking …rst-di¤erences of (20.3) eliminates the individual-speci…c e¤ect: yit =

yit

1

+

However, if eit is iid, then it will be correlated with E ( yit

1

eit ) = E ((yit

1

yit

2 ) (eit

x0it + yit eit

1

eit :

(20.4)

:

1 ))

=

E (yit

1 eit 1 )

=

2 e:

So OLS on (20.4) will be inconsistent. But if there are valid instruments, then IV or GMM can be used to estimate the equation. Typically, we use lags of the dependent variable, two periods back, as yt 2 is uncorrelated with eit : Thus values of yit k ; k 2, are valid instruments. Hence a valid estimator of and is to estimate (20.4) by IV using yt 2 as an instrument for yt 1 (which is just identi…ed). Alternatively, GMM using yt 2 and yt 3 as instruments (which is overidenti…ed, but loses a time-series observation). A more sophisticated GMM estimator recognizes that for time-periods later in the sample, there are more instruments available, so the instrument list should be di¤erent for each equation. This is conveniently organized by the GMM principle, as this enables the moments from the di¤erent timeperiods to be stacked together to create a list of all the moment conditions. A simple application of GMM yields the parameter estimates and standard errors.

Chapter 21

Nonparametric Density Estimation 21.1

Kernel Density Estimation

d F (x): Let X be a random variable with continuous distribution F (x) and density f (x) = dx The goal is to estimate f (x) from a random sample (X ; :::; X g While F (x) can be estimated by 1 n P d ^ the EDF F^ (x) = n 1 ni=1 1 (Xi x) ; we cannot de…ne dx F (x) since F^ (x) is a step function. The standard nonparametric method to estimate f (x) is based on smoothing using a kernel. While we are typically interested in estimating the entire function f (x); we can simply focus on the problem where x is a speci…c …xed number, and then see how the method generalizes to estimating the entire function.

De…nition 21.1.1 K(u) is a second-order kernel function if it is a symmetric zero-mean density function.

Three common choices for kernels include the Normal u2 2

1 K(u) = p exp 2 the Epanechnikov K(u) =

3 4

1

15 16

1

u2 ; juj 1 0 juj > 1

and the Biweight or Quartic K(u) =

u2 0

2

; juj 1 juj > 1

In practice, the choice between these three rarely makes a meaningful di¤erence in the estimates. The kernel functions are used to smooth the data. The amount of smoothing is controlled by the bandwidth h > 0. Let 1 u Kh (u) = K : h h be the kernel K rescaled by the bandwidth h: The kernel density estimator of f (x) is n

1X Kh (Xi f^(x) = n i=1

289

x) :

290

CHAPTER 21. NONPARAMETRIC DENSITY ESTIMATION

This estimator is the average of a set of weights. If a large number of the observations Xi are near x; then the weights are relatively large and f^(x) is larger. Conversely, if only a few Xi are near x; then the weights are small and f^(x) is small. The bandwidth h controls the meaning of “near”. Interestingly, f^(x) is a valid density. That is, f^(x) 0 for all x; and Z

1

f^(x)dx =

1

Z

1 1

n

n

1X Kh (Xi n

1X n

x) dx =

i=1

i=1

Z

n

1

Kh (Xi

x) dx =

1

1X n i=1

where the second-to-last equality makes the change-of-variables u = (Xi We can also calculate the moments of the density f^(x): The mean is Z

1

n

1X n

xf^(x)dx =

1

1 n

=

1 n

=

1 n

=

Z

1

xKh (Xi

1 i=1 Z n X 1 i=1 n X i=1 n X

1

K (u) du = 1

1

x)=h:

x) dx

(Xi + uh) K (u) du

1

Z

Xi

Z

n

1X K (u) du + h n 1

1

i=1

Z

1

uK (u) du

1

Xi

i=1

the sample mean of the Xi ; where the second-to-last equality used the change-of-variables u = (Xi x)=h which has Jacobian h: The second moment of the estimated density is Z

1

n

x f^(x)dx = 2

1

= = =

1X n 1 n 1 n 1 n

Z

1

1 i=1 Z n 1 X i=1 n X i=1 n X

x2 Kh (Xi

x) dx

(Xi + uh)2 K (u) du

1

n

Xi2 +

2X Xi h n i=1

Xi2 + h2

Z

1

n

K(u)du +

1

1X 2 h n i=1

Z

1

u2 K (u) du

1

2 K

i=1

where 2 K

=

Z

1

u2 K (u) du

1

is the variance of the kernel. It follows that the variance of the density f^(x) is Z

1 1

x2 f^(x)dx

Z

1 1

xf^(x)dx

n

2

=

1X 2 X i + h2 n

n

2 K

i=1

2

= ^ +

h2 2K

Thus the variance of the estimated density is in‡ated by the factor h2 moment.

1X Xi n i=1

2 K

!2

relative to the sample

291

CHAPTER 21. NONPARAMETRIC DENSITY ESTIMATION

21.2

Asymptotic MSE for Kernel Estimates

For …xed x and bandwidth h observe that Z Z 1 Z 1 Kh (uh) f (x + hu)hdu = Kh (z x) f (z)dz = EKh (X x) = 1

1

1

K (u) f (x + hu)du

1

The second equality uses the change-of variables u = (z x)=h: The last expression shows that the expected value is an average of f (z) locally about x: This integral (typically) is not analytically solvable, so we approximate it using a second order Taylor expansion of f (x + hu) in the argument hu about hu = 0; which is valid as h ! 0: Thus 1 f (x + hu) ' f (x) + f 0 (x)hu + f 00 (x)h2 u2 2 and therefore EKh (X

Z

1

1 K (u) f (x) + f 0 (x)hu + f 00 (x)h2 u2 du 2 1 Z Z 1 Z 1 1 = f (x) K (u) du + f 0 (x)h K (u) udu + f 00 (x)h2 2 1 1 1 = f (x) + f 00 (x)h2 2K : 2

x) '

1

K (u) u2 du

1

The bias of f^(x) is then n

1X EKh (Xi f (x) = n

Bias(x) = Ef^(x)

x)

i=1

1 f (x) = f 00 (x)h2 2

2 K:

We see that the bias of f^(x) at x depends on the second derivative f 00 (x): The sharper the derivative, the greater the bias. Intuitively, the estimator f^(x) smooths data local to Xi = x; so is estimating a smoothed version of f (x): The bias results from this smoothing, and is larger the greater the curvature in f (x): We now examine the variance of f^(x): Since it is an average of iid random variables, using …rst-order Taylor approximations and the fact that n 1 is of smaller order than (nh) 1 var (x) = = ' = ' =

1 var (Kh (Xi x)) n 1 1 EKh (Xi x)2 (EKh (Xi x))2 n n Z 1 1 z x 2 1 K f (x)2 f (z)dz nh2 1 h n Z 1 1 K (u)2 f (x + hu) du nh 1 Z f (x) 1 K (u)2 du nh 1 f (x) R(K) : nh

R1 where R(K) = 1 K (u)2 du is called the roughness of K: Together, the asymptotic mean-squared error (AMSE) for …xed x is the sum of the approximate squared bias and approximate variance 1 AM SEh (x) = f 00 (x)2 h4 4

4 K

+

f (x) R(K) : nh

CHAPTER 21. NONPARAMETRIC DENSITY ESTIMATION

292

A global measure of precision is the asymptotic mean integrated squared error (AMISE) Z h4 4K R(f 00 ) R(K) + : (21.1) AM ISEh = AM SEh (x)dx = 4 nh R where R(f 00 ) = (f 00 (x))2 dx is the roughness of f 00 : Notice that the …rst term (the squared bias) is increasing in h and the second term (the variance) is decreasing in nh: Thus for the AMISE to decline with n; we need h ! 0 but nh ! 1: That is, h must tend to zero, but at a slower rate than n 1 : Equation (21.1) is an asymptotic approximation to the MSE. We de…ne the asymptotically optimal bandwidth h0 as the value which minimizes this approximate MSE. That is, h0 = argmin AM ISEh h

It can be found by solving the …rst order condition d AM ISEh = h3 dh

R(K) =0 nh2

00 4 K R(f )

yielding h0 =

R(K) 4 R(f 00 ) K

1=5

n

1=2

:

(21.2)

This solution takes the form h0 = cn 1=5 where c is a function of K and f; but not of n: We thus say that the optimal bandwidth is of order O(n 1=5 ): Note that this h declines to zero, but at a very slow rate. In practice, how should the bandwidth be selected? This is a di¢cult problem, and there is a large and continuing literature on the subject. The asymptotically optimal choice given in (21.2) depends on R(K); 2K ; and R(f 00 ): The …rst two are determined by the kernel function. Their values for the three functions introduced in the previous section are given here. R R1 2 = 1 u2 K (u) du K R(K) = 1 K (u)2 du K 1 p Gaussian 1 1/(2 ) Epanechnikov 1=5 1=5 Biweight 1=7 5=7 An obvious di¢culty is that R(f 00 ) is unknown. A classic simple solution proposed by Silverman (1986)has come to be known as the reference bandwidth or Silverman’s Rule-of-Thumb. It uses formula (21.2) but replaces R(f 00 ) with ^ 5 R( 00 ); where is the N(0; 1) distribution and ^ 2 is an estimate of 2 = var(X): This choice for h gives an optimal rule when f (x) is normal, and gives a nearly optimal rule when f (x) is close to normal. The downside is that if the density is very far p from normal, the rule-of-thumb h can be quite ine¢cient. We can calculate that R( 00 ) = 3= (8 ) : Together with the above table, we …nd the reference rules for the three kernel functions introduced earlier. Gaussian Kernel: hrule = 1:06^ n 1=5 Epanechnikov Kernel: hrule = 2:34^ n 1=5 Biweight (Quartic) Kernel: hrule = 2:78^ n 1=5 Unless you delve more deeply into kernel estimation methods the rule-of-thumb bandwidth is a good practical bandwidth choice, perhaps adjusted by visual inspection of the resulting estimate f^(x): There are other approaches, but implementation can be delicate. I now discuss some of these choices. The plug-in approach is to estimate R(f 00 ) in a …rst step, and then plug this estimate into the formula (21.2). This is more treacherous than may …rst appear, as the optimal h for estimation of the roughness R(f 00 ) is quite di¤erent than the optimal h for estimation of f (x): However, there

CHAPTER 21. NONPARAMETRIC DENSITY ESTIMATION

293

are modern versions of this estimator work well, in particular the iterative method of Sheather and Jones (1991). Another popular choice for selection of h is cross-validation. This works by constructing an estimate of the MISE using leave-one-out estimators. There are some desirable properties of cross-validation bandwidths, but they are also known to converge very slowly to the optimal values. They are also quite ill-behaved when the data has some discretization (as is common in economics), in which case the cross-validation rule can sometimes select very small bandwidths leading to dramatically undersmoothed estimates. Fortunately there are remedies, which are known as smoothed cross-validation which is a close cousin of the bootstrap.

Appendix A

Matrix Algebra A.1

Notation

A scalar a is a single number. A vector a is a k 1 list of numbers, typically arranged in a column. We write this as 0 1 a1 B a2 C B C a=B . C . @ . A ak

Equivalently, a vector a is an element of Euclidean k space, written as a 2 Rk : If k = 1 then a is a scalar. A matrix A is a k r rectangular array of numbers, written as 2 3 a11 a12 a1r 6 a21 a22 a2r 7 6 7 A=6 . .. .. 7 4 .. . . 5 ak1 ak2

akr

By convention aij refers to the element in the i0 th row and j 0 th column of A: If r = 1 then A is a column vector. If k = 1 then A is a row vector. If r = k = 1; then A is a scalar. A standard convention (which we will follow in this text whenever possible) is to denote scalars by lower-case italics (a); vectors by lower-case bold italics (a); and matrices by upper-case bold italics (A): Sometimes a matrix A is denoted by the symbol (aij ): A matrix can be written as a set of column vectors or as a set of row vectors. That is, 3 2 1

A=

a1 a2

ar

6 2 7 7 6 =6 . 7 . 4 . 5 k

where

2

6 6 ai = 6 4

are column vectors and j

=

a1i a2i .. . aki

aj1 aj2 294

3 7 7 7 5 ajr

295

APPENDIX A. MATRIX ALGEBRA are row vectors. The transpose of a matrix, denoted A0 ; is obtained by Thus 2 a11 a21 ak1 6 a12 a22 ak2 6 A0 = 6 . . .. .. 4 .. . a1r a2r akr

‡ipping the matrix on its diagonal. 3 7 7 7 5

Alternatively, letting B = A0 ; then bij = aji . Note that if A is k r, then A0 is r k: If a is a k 1 vector, then a0 is a 1 k row vector. An alternative notation for the transpose of A is A> : A matrix is square if k = r: A square matrix is symmetric if A = A0 ; which requires aij = aji : A square matrix is diagonal if the o¤-diagonal elements are all zero, so that aij = 0 if i 6= j: A square matrix is upper (lower) diagonal if all elements below (above) the diagonal equal zero. An important diagonal matrix is the identity matrix, which has ones on the diagonal. The k k identity matrix is denoted as 3 2 1 0 0 6 0 1 0 7 7 6 Ik = 6 . . .. 7 : . . 4 . . . 5 0 0 1 A partitioned matrix takes the form 2 A11 A12 6 A21 A22 6 A=6 . .. 4 .. . Ak1 Ak2

A1r A2r .. . Akr

where the Aij denote matrices, vectors and/or scalars.

A.2

3 7 7 7 5

Matrix Addition

If the matrices A = (aij ) and B = (bij ) are of the same order, we de…ne the sum A + B = (aij + bij ) : Matrix addition follows the communtative and associative laws: A+B = B+A A + (B + C) = (A + B) + C:

A.3

Matrix Multiplication

If A is k

r and c is real, we de…ne their product as Ac = cA = (aij c) :

If a and b are both k

1; then their inner product is a0 b = a1 b1 + a2 b2 +

+ ak bk =

k X

aj bj :

j=1

Note that a0 b = b0 a: We say that two vectors a and b are orthogonal if a0 b = 0:

296

APPENDIX A. MATRIX ALGEBRA

If A is k r and B is r s; so that the number of columns of A equals the number of rows of B; we say that A and B are conformable. In this event the matrix product AB is de…ned. Writing A as a set of row vectors and B as a set of column vectors (each of length r); then the matrix product is de…ned as 2 0 3 a1 6 a0 7 6 2 7 bs AB = 6 . 7 b1 b2 4 .. 5 a0k 2 0 3 a01 bs a1 b1 a01 b2 6 a0 b1 a0 b2 a02 bs 7 2 6 2 7 = 6 . .. .. 7 : . 4 . . . 5 a0k b1 a0k b2 a0k bs

Matrix multiplication is not communicative: in general AB 6= BA: However, it is associative and distributive: A (BC) = (AB) C A (B + C) = AB + AC An alternative way to write the matrix product is to use matrix partitions. For example, AB = =

B 11 B 12 B 21 B 22

A11 A12 A21 A22

A11 B 11 + A12 B 21 A11 B 12 + A12 B 22 A21 B 11 + A22 B 21 A21 B 12 + A22 B 22

As another example,

AB =

A1 A2

Ar

= A1 B 1 + A2 B 2 + r X = Aj B j

2 6 6 6 4

B1 B2 .. .

Br + Ar B r

:

3 7 7 7 5

j=1

An important property of the identity matrix is that if A is k r; then AI r = A and I k A = A: The k r matrix A, r k, is called orthogonal if A0 A = I r :

A.4

Trace

The trace of a k

k square matrix A is the sum of its diagonal elements tr (A) =

k X

aii :

i=1

Some straightforward properties for square matrices A and B and real c are tr (cA) = c tr (A) tr A0

= tr (A)

tr (A + B) = tr (A) + tr (B) tr (I k ) = k:

297

APPENDIX A. MATRIX ALGEBRA Also, for k

r A and r

k B we have tr (AB) = tr (BA) :

(A.1)

Indeed, 2

6 6 tr (AB) = tr 6 4 k X

=

i=1 k X

=

a01 b1 a01 b2 a02 b1 a02 b2 .. .. . . 0 0 ak b1 ak b2

a01 bk a02 bk .. . a0k bk

3 7 7 7 5

a0i bi b0i ai

i=1

= tr (BA) :

A.5

Rank and Inverse

The rank of the k

r matrix (r

k) A=

a1 a2

ar

is the number of linearly independent columns aj ; and is written as rank (A) : We say that A has full rank if rank (A) = r: A square k k matrix A is said to be nonsingular if it is has full rank, e.g. rank (A) = k: This means that there is no k 1 c 6= 0 such that Ac = 0: If a square k k matrix A is nonsingular then there exists a unique matrix k k matrix A 1 called the inverse of A which satis…es 1

AA

=A

1

A = Ik:

For non-singular A and C; some important properties include

AA A

A

1

1

1 0

= A

1

=

0

A

A = Ik 1

(AC)

1

= C

1

(A + C)

1

= A

1

A

1

+C

1

1

(A + C)

1

= A

1

A

1

+C

1

1

A

1

C

1

A

1

Also, if A is an orthogonal matrix, then A 1 = A: Another useful result for non-singular A is known as the Woodbury matrix identity (A + BCD) In particular, for C = Morrison formula

1

=A

1

A

1

BC C + CDA

1

BC

1

CDA

1

:

(A.2)

1; B = b and D = b0 for vector b we …nd what is known as the Sherman– A

bb0

1

=A

1

+ 1

b0 A

1

b

1

A

1

bb0 A

1

:

(A.3)

298

APPENDIX A. MATRIX ALGEBRA The following fact about inverting partitioned matrices is quite useful. A11 A12 A21 A22

1

=

A11 A12 A21 A22

=

A1112 A2211 A21 A111

A1112 A12 A221 A2211

(A.4)

where A11 2 = A11 A12 A221 A21 and A22 1 = A22 A21 A111 A12 : There are alternative algebraic representations for the components. For example, using the Woodbury matrix identity you can show the following alternative expressions A11 = A111 + A111 A12 A2211 A21 A111 A22 = A221 + A221 A21 A1112 A12 A221 A12 =

A111 A12 A2211

A21 =

1 A221 A21 A112

Even if a matrix A does not possess an inverse, we can still de…ne the Moore-Penrose generalized inverse A as the matrix which satis…es AA A = A A AA = A AA is symmetric A A is symmetric For any matrix A; the Moore-Penrose generalized inverse A exists and is unique. For example, if A11 0 A= 0 0 then A =

A.6

A11 0 0 0

:

Determinant

The determinant is a measure of the volume of a square matrix. While the determinant is widely used, its precise de…nition is rarely needed. However, we present the de…nition here for completeness. Let A = (aij ) be a general k k matrix . Let = (j1 ; :::; jk ) denote a permutation of (1; :::; k) : There are k! such permutations. There is a unique count of the number of inversions of the indices of such permutations (relative to the natural order (1; :::; k) ; and let " = +1 if this count is even and " = 1 if the count is odd. Then the determinant of A is de…ned as X det A = " a1j1 a2j2 akjk : For example, if A is 2 2; then the two permutations of (1; 2) are (1; 2) and (2; 1) ; for which "(1;2) = 1 and "(2;1) = 1. Thus det A = "(1;2) a11 a22 + "(2;1) a21 a12 = a11 a22 Some properties include det (A) = det (A0 ) det (cA) = ck det A

a12 a21 :

299

APPENDIX A. MATRIX ALGEBRA det (AB) = (det A) (det B) det A

1

= (det A)

A B C D

det

1

= (det D) det A

1

BD

C if det D 6= 0

det A 6= 0 if and only if A is nonsingular. If A is triangular (upper or lower), then det A = If A is orthogonal, then det A =

A.7

1

Qk

i=1 aii

Eigenvalues

The characteristic equation of a k

k square matrix A is det (A

I k ) = 0:

The left side is a polynomial of degree k in so it has exactly k roots, which are not necessarily distinct and may be real or complex. They are called the latent roots or characteristic roots or eigenvalues of A. If i is an eigenvalue of A; then A i I k is singular so there exists a non-zero vector hi such that (A i I k ) hi = 0: The vector hi is called a latent vector or characteristic vector or eigenvector of A corresponding to i : We now state some useful properties. Let i and hi , i = 1; :::; k denote the k eigenvalues and eigenvectors of a square matrix A: Let be a diagonal matrix with the characteristic roots in the diagonal, and let H = [h1 hk ]: Q det(A) = ki=1 i P tr(A) = ki=1 i

A is non-singular if and only if all its characteristic roots are non-zero. If A has distinct characteristic roots, there exists a nonsingular matrix P such that A = P 1 P and P AP 1 = . If A is symmetric, then A = H H 0 and H 0 AH = ; and the characteristic roots are all real. A = H H 0 is called the spectral decomposition of a matrix. When the eigenvalues of k k A are real they are written in decending order k : We also write min (A) = k = minf ` g and max (A) = 1 = maxf ` g. The characteristic roots of A

1

are

1

1

;

2

1

; ...,

1 k

:

The matrix H has the orthonormal properties H 0 H = I and HH 0 = I. H

1

= H 0 and (H 0 )

1

=H

1

2

300

APPENDIX A. MATRIX ALGEBRA

A.8

Positive De…niteness

We say that a k k symmetric square matrix A is positive semi-de…nite if for all c 6= 0; 0: This is written as A 0: We say that A is positive de…nite if for all c 6= 0; c0 Ac > 0: This is written as A > 0: Some properties include: c0 Ac

If A = G0 G for some matrix G; then A is positive semi-de…nite. (For any c 6= 0; c0 Ac = 0 0 where = Gc:) If G has full rank, then A is positive de…nite. If A is positive de…nite, then A is non-singular and A

1

exists. Furthermore, A

1

> 0:

A > 0 if and only if it is symmetric and all its characteristic roots are positive. By the spectral decomposition, A = H H 0 where H 0 H = I and is diagonal with nonnegative diagonal elements. All diagonal elements of are strictly positive if (and only if) A > 0: If A > 0 then A

1

=H

1

H 0:

If A 0 and rank (A) = r < k then A = H H 0 where A 1 1 1 generalized inverse, and = diag 1 ; 2 ; :::; k ; 0; :::; 0

is the Moore-Penrose

If A 0 we can …nd a matrix B such that A = BB 0 : We call B a matrix square root of A: The matrix B need not be unique. One way to construct B is to use the spectral decomposition A = H H 0 where is diagonal, and then set B = H 1=2 : There is a unique root root B which is also positive semi-de…nite B 0: A square matrix A is idempotent if AA = A: If A is idempotent and symmetric then all its characteristic roots equal either zero or one and is thus positive semi-de…nite. To see this, note that we can write A = H H 0 where H is orthogonal and contains the r (real) characteristic roots. Then A = AA = H H 0 H H 0 = H 2 H 0 : By the uniqueness of the characteristic roots, we deduce that 2 = and 2i = i for i = 1; :::; r: Hence they must equal either 0 or 1. It follows that the spectral decomposition of idempotent A takes the form Ik r 0 H0 (A.5) A=H 0 0 with H 0 H = I k . Additionally, tr(A) = rank(A): If A is idempotent then I A is also idempotent. One useful fact is that A is idempotent then for any conformable vector c,

0

c (I

c0 Ac

c0 c

(A.6)

A) c

0

(A.7)

cc

To see this, note that c0 c = c0 Ac + c0 (I

A) c:

Since A and I A are idempotent, they are both positive semi-de…nite, so both c0 Ac and 0 c (I A) c are negative. Thus they must satisfy (A.6)-(A.7),.

301

APPENDIX A. MATRIX ALGEBRA

A.9

Matrix Calculus 1 and g(x) = g(x1 ; :::; xk ) : Rk ! R: The vector derivative is 1 0 @ @x1 g (x) @ C B .. g (x) = @ A . @x @ @xk g (x)

Let x = (x1 ; :::; xk ) be k

and

@ g (x) = @x0 Some properties are now summarized. @ @x @ @x0 @ @x

(a0 x) =

@ @xk g (x)

:

(x0 a) = a

(Ax) = A (x0 Ax) = (A + A0 ) x

@2 @x@x0

A.10

@ @x

@ @x1 g (x)

(x0 Ax) = A + A0

Kronecker Products and the Vec Operator

Let A = [a1 a2

an ] be m

n: The vec of A; denoted by vec (A) ; is the mn 1 0 a1 B a2 C C B vec (A) = B . C : @ .. A

1 vector

an

Let A = (aij ) be an m n matrix and let B be any matrix. The Kronecker product of A and B; denoted A B; is the matrix 3 2 a11 B a12 B a1n B 6 a21 B a22 B a2n B 7 7 6 A B=6 7: .. .. .. 5 4 . . . am1 B am2 B amn B

Some important properties are now summarized. These results hold for matrices for which all matrix multiplications are conformable. (A + B) (A A

C=A

B) (C (B

(A tr (A

D) = AC

C) = (A

B)0 = A0

C

BD

B)

C

B0

B) = tr (A) tr (B)

If A is m (A

C +B

B)

m and B is n 1

=A

1

B) = (det (A))n (det (B))m

1

B

If A > 0 and B > 0 then A vec (ABC) = (C 0

n; det(A

B>0

A) vec (B) 0

tr (ABCD) = vec (D 0 ) (C 0

A) vec (B)

302

APPENDIX A. MATRIX ALGEBRA

A.11

Vector and Matrix Norms

The Euclidean norm of an m

1 vector a is a0 a

kak =

m X

=

1=2

a2i

i=1

The Euclidean norm of an m

!1=2

:

n matrix A is kAk = kvec (A)k

1=2

tr A0 A 0 11=2 m X n X = @ a2ij A :

=

i=1 j=1

If an m

m matrix A is symmetric with eigenvalues m X

kAk =

`=1

2 `

`;

!1=2

` = 1; :::; m; then

:

To see this, by the spectral decomposition A = H H 0 with H 0 H = I and = diagf so !1=2 m X 1=2 0 0 1=2 2 kAk = tr H H H H = (tr ( )) = : `

1 ; :::;

m g;

(A.8)

`=1

A useful calculation is for any m

1 vectors a and b, using (A.1), 0 1=2

ab0 = tr ba0 ab and in particular

A.12

= b0 ba0 a

1=2

= kak kbk

aa0 = kak2

(A.9)

Matrix Inequalities

Schwarz Inequality: For any m

1 vectors a and b, a0 b

Schwarz Matrix Inequality: For any m

kak kbk : n matrices A and B;

A0 B Triangle Inequality: For any m

kAk kBk :

(A.11)

n matrices A and B; kA + Bk

Trace Inequality. For any m

(A.10)

kAk + kBk :

m matrices A and B such that A is symmetric and B tr (AB)

max (A) tr (B)

(A.12) 0 (A.13)

303

APPENDIX A. MATRIX ALGEBRA where max (A) is the largest eigenvalue of A. Quadratic Inequality. For any m 1 b and m b0 Ab Norm Inequality. For any m

0 max (A) b b

(A.14)

m matrices A and B such that A kABk

min (B)

` (AB)

0 and B

0

max (A) kBk

where max (A) is the largest eigenvalue of A. Eigenvalue Product Inequaly. For any k ` (AB) are real and satisfy min (A)

m symmetric matrix A

=

`

(A.15)

k matrices A

0 and B

A1=2 BA1=2

max (A)

0; the eigenvalues

max (B)

(A.16)

(Zhang and Zhang, 2006, Corollary 11) Jensen’s Inequality. If g( ) : R ! R is convex, then for any non-negative weights aj such that P m j=1 aj = 1; and any real numbers xj 0 1 m X g@ aj xj A

m X

j=1

In particular, setting aj = 1=m; then 0

aj g (xj ) :

(A.17)

j=1

1 m X 1 xj A g@ m

m

1 X g (xj ) : m

j=1

(A.18)

j=1

Loève’s cr Inequality. For r > 0; m X

r

aj

cr

j=1

m X

jaj jr

(A.19)

2a0 a + 2b0 b

(A.20)

j=1

where cr = 1 when r 1 and cr = mr 1 when r c2 Inequality. For any m 1 vectors a and b,

1:

(a + b)0 (a + b)

Proof of Schwarz Inequality: First, suppose that kbk = 0: Then b = 0 and both ja0 bj = 0 and 1 0 kak kbk = 0 so the inequality is true. Second, suppose that kbk > 0 and de…ne c = a b b0 b b a: Since c is a vector, c0 c 0: Thus 0

c0 c = a0 a

Rearranging, this implies that a0 b

2

a0 b a0 a

Taking the square root of each side yields the result.

2

= b0 b :

b0 b :

304

APPENDIX A. MATRIX ALGEBRA

Proof of Schwarz Matrix Inequality: Partition A = [a1 ; :::; an ] and B = [b1 ; :::; bn ]. Then by partitioned matrix multiplication, the de…nition of the matrix Euclidean norm and the Schwarz inequality A0 B

a01 b1 a01 b2 a02 b1 a02 b2 .. .. . .

=

..

.

ka1 k kb1 k ka1 k kb2 k ka2 k kb1 k ka2 k kb2 k .. .. .. . . . 0 11=2 n X n X @ = kai k2 kbj k2 A i=1 j=1

=

n X i=1

2

kai k

!1=2

n X i=1

2

kbi k

!1=2

11=2 0 11=2 0 n X m n X m X X = @ a2ji A @ kbji k2 A i=1 j=1

i=1 j=1

= kAk kBk

Proof of Triangle Inequality: Let a = vec (A) and b = vec (B) . Then by the de…nition of the matrix norm and the Schwarz Inequality kA + Bk2 = ka + bk2

= a0 a + 2a0 b + b0 b a0 a + 2 a0 b + b0 b kak2 + 2 kak kbk + kbk2

= (kak + kbk)2

= (kAk + kBk)2

Proof of Trace Inequality. By the spectral decomposition for symmetric matices, A = H H 0 where has the eigenvalues j of A on the diagonal and H is orthonormal. De…ne C = H 0 BH which has non-negative diagonal elements Cjj since B is positive semi-de…nite. Then tr (AB) = tr ( C) =

m X

j Cjj

j=1

where the inequality uses the fact that Cjj

max j

j

m X

Cjj =

max (A) tr (C)

j=1

0: But note that

tr (C) = tr H 0 BH = tr HH 0 B = tr (B) since H is orthonormal. Thus tr (AB)

max (A) tr (B)

as stated.

Proof of Quadratic Inequality: In the Trace Inequality set B = bb0 and note tr (AB) = b0 Ab and tr (B) = b0 b:

305

APPENDIX A. MATRIX ALGEBRA Proof of Norm Inequality. Using the Trace Inequality kABk = (tr (BAAB))1=2 = (tr (AABB))1=2 (

max (AA) tr (BB))

1=2

max (A) kBk :

=

The …nal equality holds since A 0 implies that max (AA) = max (A)2 : Proof of Jensen’s Inequality (A.17). By the de…nition of convexity, for any g ( x1 + (1 This implies

) x2 )

g (x1 ) + (1

0 1 0 m X aj xj A = g @a1 g (x1 ) + (1 g@ j=1

a1 g (x1 )+(1

a1 ) and 0

Pm

j=2 bj

a1 ) @b2 g(x2 ) + (1

where cj = bj =(1

a1 )

m X

0

a1 ) g @

a1 g (x1 ) + (1 where bj = aj =(1

) g (x2 ) :

j=2

m X j=2

aj 1

a1 1

2 [0; 1] (A.21)

1

xj A

bj xj A :

= 1: By another application of (A.21) this is bounded by 1 0 11 0 m m X X cj xj A cj xj AA = a1 g (x1 )+a2 g(x2 )+(1 a1 ) (1 b2 )g @ b2 )g @ j=2

j=2

b2 ): By repeated application of (A.21) we obtain (A.17).

Proof of Loève’s cr Inequality. For r 1 this is simply a rewriting of the …nite form Jensen’s Pm inequality (A.18) with g(u) = ur : For r < 1; de…ne bj = jaj j = bj 1 j=1 jaj j : The facts that 0 r and r < 1 imply bj bj and thus m m X X 1= bj brj j=1

which implies

j=1

0 1r m X @ jaj jA

m X

j=1

The proof is completed by observing that 0 1r m X @ aj A

j=1

0 @

j=1

m X j=1

jaj jr : 1r

jaj jA :

Proof of c2 Inequality. By the cr inequality, (aj + bj )2 (a + b)0 (a + b) =

m X

2a2j + 2b2j . Thus

(aj + bj )2

j=1 m X

2

j=1 0

a2j

+2

m X j=1

0

= 2a a + 2b b

b2j

Appendix B

Probability B.1

Foundations

The set S of all possible outcomes of an experiment is called the sample space for the experiment. Take the simple example of tossing a coin. There are two outcomes, heads and tails, so we can write S = fH; T g: If two coins are tossed in sequence, we can write the four outcomes as S = fHH; HT; T H; T T g: An event A is any collection of possible outcomes of an experiment. An event is a subset of S; including S itself and the null set ;: Continuing the two coin example, one event is A = fHH; HT g; the event that the …rst coin is heads. We say that A and B are disjoint or mutually exclusive if A \ B = ;: For example, the sets fHH; HT g and fT Hg are disjoint. Furthermore, if the sets A1 ; A2 ; ::: are pairwise disjoint and [1 i=1 Ai = S; then the collection A1 ; A2 ; ::: is called a partition of S: The following are elementary set operations: Union: A [ B = fx : x 2 A or x 2 Bg: Intersection: A \ B = fx : x 2 A and x 2 Bg: Complement: Ac = fx : x 2 = Ag: The following are useful properties of set operations. Communtatitivity: A [ B = B [ A; A \ B = B \ A: Associativity: A [ (B [ C) = (A [ B) [ C; A \ (B \ C) = (A \ B) \ C: Distributive Laws: A \ (B [ C) = (A \ B) [ (A \ C) ; A [ (B \ C) = (A [ B) \ (A [ C) : c c c c DeMorgan’s Laws: (A [ B) = A \ B ; (A \ B) = Ac [ B c : A probability function assigns probabilities (numbers between 0 and 1) to events A in S: This is straightforward when S is countable; when S is uncountable we must be somewhat more careful: A set B is called a sigma algebra (or Borel …eld) if ; 2 B , A 2 B implies Ac 2 B, and A1 ; A2 ; ::: 2 B implies [1 i=1 Ai 2 B. A simple example is f;; Sg which is known as the trivial sigma algebra. For any sample space S; let B be the smallest sigma algebra which contains all of the open sets in S: When S is countable, B is simply the collection of all subsets of S; including ; and S: When S is the real line, then B is the collection of all open and closed intervals. We call B the sigma algebra associated with S: We only de…ne probabilities for events contained in B. We now can give the axiomatic de…nition of probability. Given S and B, a probability function Pr satis…es Pr(S) 0 for all A 2 B, and if A1 ; A2 ; ::: 2 B are pairwise disjoint, then P1= 1; Pr(A) Pr(A ): Pr ([1 A ) = i i i=1 i=1 Some important properties of the probability function include the following Pr (;) = 0 Pr(A)

1

Pr (Ac ) = 1

Pr(A) 306

307

APPENDIX B. PROBABILITY Pr (B \ Ac ) = Pr(B)

Pr(A \ B)

Pr (A [ B) = Pr(A) + Pr(B) If A

B then Pr(A)

Pr(A \ B)

Pr(B)

Bonferroni’s Inequality: Pr(A \ B) Boole’s Inequality: Pr (A [ B)

Pr(A) + Pr(B)

1

Pr(A) + Pr(B)

For some elementary probability models, it is useful to have simple rules to count the number of objects in a set. These counting rules are facilitated by using the binomial coe¢cients which are de…ned for nonnegative integers n and r; n r; as n r

=

n! : r! (n r)!

When counting the number of objects in a set, there are two important distinctions. Counting may be with replacement or without replacement. Counting may be ordered or unordered. For example, consider a lottery where you pick six numbers from the set 1, 2, ..., 49. This selection is without replacement if you are not allowed to select the same number twice, and is with replacement if this is allowed. Counting is ordered or not depending on whether the sequential order of the numbers is relevant to winning the lottery. Depending on these two distinctions, we have four expressions for the number of objects (possible arrangements) of size r from n objects. Without Replacement n! (n r)! n r

Ordered Unordered

With Replacement nr n+r 1 r

In the lottery example, if counting is unordered and without replacement, the number of potential combinations is 49 6 = 13; 983; 816. If Pr(B) > 0 the conditional probability of the event A given the event B is Pr (A j B) =

Pr (A \ B) : Pr(B)

For any B; the conditional probability function is a valid probability function where S has been replaced by B: Rearranging the de…nition, we can write Pr(A \ B) = Pr (A j B) Pr(B) which is often quite useful. We can say that the occurrence of B has no information about the likelihood of event A when Pr (A j B) = Pr(A); in which case we …nd Pr(A \ B) = Pr (A) Pr(B)

(B.1)

We say that the events A and B are statistically independent when (B.1) holds. Furthermore, we say that the collection of events A1 ; :::; Ak are mutually independent when for any subset fAi : i 2 Ig; ! Y \ Pr (Ai ) : Ai = Pr i2I

i2I

Theorem 3 (Bayes’ Rule). For any set B and any partition A1 ; A2 ; ::: of the sample space, then for each i = 1; 2; ::: Pr (B j Ai ) Pr(Ai ) Pr (Ai j B) = P1 j=1 Pr (B j Aj ) Pr(Aj )

308

APPENDIX B. PROBABILITY

B.2

Random Variables

A random variable X is a function from a sample space S into the real line. This induces a new sample space – the real line – and a new probability function on the real line. Typically, we denote random variables by uppercase letters such as X; and use lower case letters such as x for potential values and realized values. (This is in contrast to the notation adopted for most of the textbook.) For a random variable X we de…ne its cumulative distribution function (CDF) as F (x) = Pr (X

x) :

(B.2)

Sometimes we write this as FX (x) to denote that it is the CDF of X: A function F (x) is a CDF if and only if the following three properties hold: 1. limx!

1 F (x)

= 0 and limx!1 F (x) = 1

2. F (x) is nondecreasing in x 3. F (x) is right-continuous We say that the random variable X is discrete if F (x) is a step function. In the latter case, the range of X consists of a countable set of real numbers 1 ; :::; r : The probability function for X takes the form Pr (X = j ) = j ; j = 1; :::; r (B.3) Pr where 0 1 and j=1 j = 1. j We say that the random variable X is continuous if F (x) is continuous in x: In this case Pr(X = ) = 0 for all 2 R so the representation (B.3) is unavailable. Instead, we represent the relative probabilities by the probability density function (PDF) f (x) = so that F (x) = and Pr (a

X

Z

d F (x) dx x

f (u)du 1

b) =

Z

b

f (u)du:

a

These expressions only make sense if F (x) is di¤erentiable. While there are examples of continuous random variables which do not possess a PDF, these cases are unusualRand are typically ignored. 1 A function f (x) is a PDF if and only if f (x) 0 for all x 2 R and 1 f (x)dx:

B.3

Expectation

For any measurable real function g; we de…ne the mean or expectation Eg(X) as follows. If X is discrete, r X Eg(X) = g( j ) j ; j=1

and if X is continuous

Eg(X) = The latter is well de…ned and …nite if Z

1 1

Z

1

g(x)f (x)dx:

1

jg(x)j f (x)dx < 1:

(B.4)

309

APPENDIX B. PROBABILITY If (B.4) does not hold, evaluate I1 =

Z

g(x)f (x)dx

g(x)>0

I2 =

Z

g(x)f (x)dx

g(x) 0; we de…ne the m0 th moment of X as EX m and the m0 th central moment as E (X EX)m : 2 : We Two special moments are the mean = EX and variance 2 = E (X )2 = EX 2 p 2 2 the standard deviation of X: We can also write = var(X). For example, this call = 2 allows the convenient expression var(a + bX) = b var(X): The moment generating function (MGF) of X is M ( ) = E exp ( X) : The MGF does not necessarily exist. However, when it does and E jXjm < 1 then dm M( ) d m

= E (X m ) =0

which is why it is called the moment generating function. More generally, the characteristic function (CF) of X is

where i =

p

C( ) = E exp (i X) 1 is the imaginary unit. The CF always exists, and when E jXjm < 1 dm C( ) d m

The Lp norm, p

= im E (X m ) : =0

1; of the random variable X is kXkp = (E jXjp )1=p :

B.4

Gamma Function

The gamma function is de…ned for

> 0 as Z 1 ( )= x

1

exp ( x) :

0

It satis…es the property

(1 + ) = ( ) so for positive integers n; (n) = (n

1)!

Special values include (1) = 1 and

1 = 1=2 : 2 Sterling’s formula is an expansion for the its logarithm log ( ) =

1 log(2 ) + 2

1 2

log

z+

1 12

1 360

3

+

1 1260

5

+

310

APPENDIX B. PROBABILITY

B.5

Common Distributions

For reference, we now list some important discrete distribution function. Bernoulli Pr (X = x) = px (1

p)1

x

;

x = 0; 1;

x

;

x = 0; 1; :::; n;

0

p

1

EX = p var(X) = p(1

p)

Binomial n x p (1 x EX = np

p)n

Pr (X = x) =

var(X) = np(1

0

p

p

1

1

p)

Geometric Pr (X = x) = p(1 p)x 1 EX = p 1 p var(X) = p2

1

;

x = 1; 2; :::;

0

Multinomial n! px 1 p x 2 x1 !x2 ! xm ! 1 2 = n;

pxmm ;

Pr (X1 = x1 ; X2 = x2 ; :::; Xm = xm ) = x1 +

+ xm

p1 +

+ pm = 1 EXi = pi var(Xi ) = npi (1

cov (Xi ; Xj ) =

pi )

npi pj

Negative Binomial Pr (X = x) = EX = var(X) =

(r + x) r p (1 x! (r) r (1 p) p r (1 p) p2

p)x

1

;

x = 0; 1; 2; :::;

0

Poisson Pr (X = x) = EX =

exp (

) x!

x

;

x = 0; 1; 2; :::;

var(X) = We now list some important continuous distributions.

>0

p

1

311

APPENDIX B. PROBABILITY Beta ( + ) x ( ) ( )

f (x) = =

1

(1

1

x)

;

0

x

1;

> 0;

>0

+

var(X) =

+ 1) ( + )2

( +

Cauchy 1 ; (1 + x2 ) EX = 1

10

2

Pareto f (x) =

x

EX =

+1

x < 1;

;

1

;

> 0;

>1 2

var(X) =

(

1)2 (

2)

;

>2

Uniform f (x) = EX = var(X) =

1

; b a a+b 2 (b a)2 12

a

x

b

>0

312

APPENDIX B. PROBABILITY Weibull f (x) =

x

EX =

1=

var(X) =

2=

1

x

exp

;

x < 1;

0

> 0;

>0

1

1+

1+

2

2

1+

;

0

1

Gamma f (x) =

1 ( )

1

x

x

exp

x < 1;

> 0;

>0

EX = 2

var(X) = Chi-Square

1 xr=2 (r=2)2r=2

f (x) =

1

exp

x ; 2

0

x < 1;

r>0

EX = r var(X) = 2r Normal f (x) =

1 p 2

exp

)2

(x 2

2

!

1 < x < 1;

;

1
0

EX = var(X) =

2

Student t f (x) =

p

r+1 2

x2 1+ r

( r+1 2 )

B.6

1 < x < 1;

;

r r 2 EX = 0 if r > 1 r var(X) = if r > 2 r 2

r>0

Multivariate Random Variables

A pair of bivariate random variables (X; Y ) is a function from the sample space into R2 : The joint CDF of (X; Y ) is F (x; y) = Pr (X x; Y y) : If F is continuous, the joint probability density function is f (x; y) =

@2 F (x; y): @x@y

For a Borel measurable set A 2 R2 ; Pr ((X; Y ) 2 A) =

Z Z

A

f (x; y)dxdy

313

APPENDIX B. PROBABILITY For any measurable function g(x; y); Eg(X; Y ) =

Z

1 1

Z

1

g(x; y)f (x; y)dxdy:

1

The marginal distribution of X is FX (x) = Pr(X = =

x)

lim F (x; y)

y!1 Z x Z 1 1

f (x; y)dydx

1

so the marginal density of X is fX (x) =

d FX (x) = dx

Z

1

f (x; y)dy:

1

Similarly, the marginal density of Y is fY (y) =

Z

1

f (x; y)dx:

1

The random variables X and Y are de…ned to be independent if f (x; y) = fX (x)fY (y): Furthermore, X and Y are independent if and only if there exist functions g(x) and h(y) such that f (x; y) = g(x)h(y): If X and Y are independent, then Z Z E (g(X)h(Y )) = g(x)h(y)f (y; x)dydx Z Z = g(x)h(y)fY (y)fX (x)dydx Z Z = g(x)fX (x)dx h(y)fY (y)dy = Eg (X) Eh (Y ) :

(B.5)

if the expectations exist. For example, if X and Y are independent then E(XY ) = EXEY: Another implication of (B.5) is that if X and Y are independent and Z = X + Y; then MZ ( ) = E exp ( (X + Y )) = E (exp ( X) exp ( Y )) = E exp

0

0

X E exp

Y

= MX ( )MY ( ):

(B.6)

The covariance between X and Y is cov(X; Y ) =

XY

= E ((X

EX) (Y

EY )) = EXY

The correlation between X and Y is corr (X; Y ) =

XY

=

XY X Y

:

EXEY:

314

APPENDIX B. PROBABILITY The Cauchy-Schwarz Inequality implies that j

XY j

1:

(B.7)

The correlation is a measure of linear dependence, free of units of measurement. If X and Y are independent, then XY = 0 and XY = 0: The reverse, however, is not true. For example, if EX = 0 and EX 3 = 0, then cov(X; X 2 ) = 0: A useful fact is that var (X + Y ) = var(X) + var(Y ) + 2 cov(X; Y ): An implication is that if X and Y are independent, then var (X + Y ) = var(X) + var(Y ); the variance of the sum is the sum of the variances. A k 1 random vector X = (X1 ; :::; Xk )0 is a function from S to Rk : Let x = (x1 ; :::; xk )0 denote a vector in Rk : (In this Appendix, we use bold to denote vectors. Bold capitals X are random vectors and bold lower case x are nonrandom vectors. Again, this is in distinction to the notation used in the bulk of the text) The vector X has the distribution and density functions F (x) = Pr(X x) @k F (x): f (x) = @x1 @xk For a measurable function g : Rk ! Rs ; we de…ne the expectation Z Eg(X) = g(x)f (x)dx Rk

where the symbol dx denotes dx1

dxk : In particular, we have the k

1 multivariate mean

= EX and k

k covariance matrix = E (X

) (X

= EXX 0

0

)0

If the elements of X are mutually independent, then is a diagonal matrix and ! k k X X var Xi = var (X i ) i=1

B.7

i=1

Conditional Distributions and Expectation

The conditional density of Y given X = x is de…ned as fY jX (y j x) =

f (x; y) fX (x)

315

APPENDIX B. PROBABILITY

if fX (x) > 0: One way to derive this expression from the de…nition of conditional probability is fY jX (y j x) = = = = = =

@ lim Pr (Y y j x X x + ") @y "!0 Pr (fY yg \ fx X x + "g) @ lim "!0 @y Pr(x X x + ") @ F (x + "; y) F (x; y) lim @y "!0 FX (x + ") FX (x) @ lim @y "!0

@ @x F (x

+ "; y) fX (x + ")

@2 @x@y F (x; y)

fX (x) f (x; y) : fX (x)

The conditional mean or conditional expectation is the function Z 1 yfY jX (y j x) dy: m(x) = E (Y j X = x) = 1

The conditional mean m(x) is a function, meaning that when X equals x; then the expected value of Y is m(x): Similarly, we de…ne the conditional variance of Y given X = x as 2

(x) = var (Y j X = x) = E (Y

m(x))2 j X = x

= E Y2 jX =x

m(x)2 :

Evaluated at x = X; the conditional mean m(X) and conditional variance 2 (X) are random variables, functions of X: We write this as E(Y j X) = m(X) and var (Y j X) = 2 (X): For example, if E (Y j X = x) = + 0 x; then E (Y j X) = + 0 X; a transformation of X: The following are important facts about conditional expectations. Simple Law of Iterated Expectations: E (E (Y j X)) = E (Y )

(B.8)

Proof : E (E (Y j X)) = E (m(X)) Z 1 = m(x)fX (x)dx 1 Z 1Z 1 yfY jX (y j x) fX (x)dydx = 1 1 Z 1 Z 1 yf (y; x) dydx = 1

= E(Y ):

1

Law of Iterated Expectations: E (E (Y j X; Z) j X) = E (Y j X)

(B.9)

316

APPENDIX B. PROBABILITY Conditioning Theorem. For any function g(x); E (g(X)Y j X) = g (X) E (Y j X)

(B.10)

Proof : Let h(x) = E (g(X)Y j X = x) Z 1 g(x)yfY jX (y j x) dy = 1 Z 1 = g(x) yfY jX (y j x) dy 1

= g(x)m(x)

where m(x) = E (Y j X = x) : Thus h(X) = g(X)m(X), which is the same as E (g(X)Y j X) = g (X) E (Y j X) :

B.8

Transformations

Suppose that X 2 Rk with continuous distribution function FX (x) and density fX (x): Let Y = g(X) where g(x) : Rk ! Rk is one-to-one, di¤erentiable, and invertible. Let h(y) denote the inverse of g(x). The Jacobian is @ h(y) : @y 0

J(y) = det

Consider the univariate case k = 1: If g(x) is an increasing function, then g(X) if X h(Y ); so the distribution function of Y is FY (y) = Pr (g(X) = Pr (X

Y if and only

y) h(Y ))

= FX (h(Y )) : Taking the derivative, the density of Y is fY (y) =

d d FY (y) = fX (h(Y )) h(y): dy dy

If g(x) is a decreasing function, then g(X)

Y if and only if X

FY (y) = Pr (g(X)

y)

= 1

Pr (X

= 1

FX (h(Y ))

and the density of Y is fY (y) =

h(Y ); so

fX (h(Y ))

h(Y ))

d h(y): dy

We can write these two cases jointly as fY (y) = fX (h(Y )) jJ(y)j :

(B.11)

This is known as the change-of-variables formula. This same formula (B.11) holds for k > 1; but its justi…cation requires deeper results from analysis. As one example, take the case X U [0; 1] and Y = log(X). Here, g(x) = log(x) and h(y) = exp( y) so the Jacobian is J(y) = exp(y): As the range of X is [0; 1]; that for Y is [0,1): Since fX (x) = 1 for 0 x 1 (B.11) shows that fY (y) = exp( y); an exponential density.

0

y

1;

317

APPENDIX B. PROBABILITY

B.9

Normal and Related Distributions

The standard normal density is x2 2

1 (x) = p exp 2

1 < x < 1:

;

It is conventional to write X N (0; 1) ; and to denote the standard normal density function by (x) and its distribution function by (x): The latter has no closed-form solution. The normal density has all moments …nite. Since it is symmetric about zero all odd moments are zero. By iterated integration by parts, we can also show that EX 2 = 1 and EX 4 = 3: In fact, for any positive integer m, EX 2m = (2m 1)!! = (2m 1) (2m 3) 1: Thus EX 4 = 3; EX 6 = 15; EX 8 = 105; and EX 10 = 945: If Z is standard normal and X = + Z; then using the change-of-variables formula, X has density ! (x )2 1 exp f (x) = p ; 1 < x < 1: 2 2 2 which is the univariate normal density. The mean and variance of the distribution are 2 ; and it is conventional to write X N ; 2 . For x 2 Rk ; the multivariate normal density is f (x) =

1 k=2

(2 )

1=2

det ( )

(x

exp

)0

1

(x

)

2

and

x 2 Rk :

;

The mean and covariance matrix of the distribution are and ; and it is conventional to write X N ( ; ). 0 The MGF and CF of the multivariate normal are exp 0 + 0 =2 and exp i 0 =2 ; respectively. If X 2 Rk is multivariate normal and the elements of X are mutually uncorrelated, then = diagf 2j g is a diagonal matrix. In this case the density function can be written as f (x) = =

1 (2 )k=2 k Y

j=1

exp 1

exp j

2 2 1) = 1

+

+ (xk

2 2 k) = k

2

k

1 1=2

(2 )

(x1 xj

j 2 j

2

! 2

!!

which is the product of marginal univariate normal densities. This shows that if X is multivariate normal with uncorrelated elements, then they are mutually independent. Theorem B.9.1 If X N (a + B ; B B 0 ) : Theorem B.9.2 Let X freedom, written 2r . Theorem B.9.3 If Z

N ( ; ) and Y = a + BX with B an invertible matrix, then Y N (0; I r ) : Then Q = X 0 X is distributed chi-square with r degrees of N (0; A) with A > 0; q

Theorem B.9.4 Let Z N (0; 1) and Q as student’s t with r degrees of freedom.

2 r

q; then Z 0 A

1

Z

2: q

be independent. Then Tr = Z=

p Q=r is distributed

318

APPENDIX B. PROBABILITY

Proof of Theorem B.9.1. By the change-of-variables formula, the density of Y = a + BX is ! 0 1 (y 1 Y) Y) Y (y ; y 2 Rk : exp f (y) = k=2 1=2 2 (2 ) det ( Y ) where

Y

= a+B and

Y

1=2

= B B 0 ; where we used the fact that det (B B 0 )

= det ( )1=2 det (B) :

Proof of Theorem B.9.2. First, suppose a random variable Q is distributed chi-square with r degrees of freedom. It has the MGF Z 1 1 E exp (tQ) = xr=2 1 exp (tx) exp ( x=2) dy = (1 2t) r=2 r r=2 2 0 2 R1 where the second equality uses the fact that 0 y a 1 exp ( by) dy = b a (a); which can be found by applying change-of-variables to the gamma function. Our goal is to calculate the MGF of r=2 2. Q = X 0 X and show that it equals (1 2t) P ; which will establish that Q r r 0 2 Note that we can write Q = X X = j=1 Zj where the Zj are independent N (0; 1) : The distribution of each of the Zj2 is Pr Zj2

p = 2 Pr (0 Zj y) Z py 1 x2 p exp = 2 dx 2 2 0 Z y s 1 ds s 1=2 exp = 1 1=2 2 0 2 2

y

using the change–of-variables s = x2 and the fact 1

f1 (x) = which is the h

Since (1

2t)

2 1

1 2

21=2

x

p

1 2

=

1=2

exp

: Thus the density of Zj2 is x 2

and by our above calculation has the MGF of E exp tZj2 = (1

the Zj2 ir 1=2

2t)

are mutually independent, (B.6) implies that the MGF of Q =

= (1

2t)

r=2

2 r

; which is the MGF of the

density as desired:

1=2

: Pr

2 j=1 Zj

is

Proof of Theorem B.9.3. The fact that A > 0 means that we can write A = CC 0 where C is non-singular. Then A 1 = C 10 C 1 and C

1

Z

N 0; C

1

AC

10

= N 0; C

C

1

1

CC 0 C

Thus Z 0A

1

Z = Z 0C

10

Z= C

1

Z

0

C

10

1

Z

= N (0; I q ) : 2 q:

Proof of Theorem B.9.4. Using the simple law of iterated expectations, Tr has distribution

319

APPENDIX B. PROBABILITY function

!

Z

p Q=r ( r

F (x) = Pr

= E Z

x

"

= E Pr Z = E

r

x

x

) Q r r x

Q r

!

!# Q jQ r

Thus its density is ! Q x r r !r ! Q Q x r r r

d f (x) = E dx = E =

Z

1

0

=

p

qx2 2r

1 p exp 2 r+1 2

r

r 2

x2 1+ r

r

1

q r

r 2

2r=2

q r=2

1

!

exp ( q=2) dq

( r+1 2 )

which is that of the student t with r degrees of freedom.

B.10

Inequalities

Jensen’s Inequality. If g( ) : Rm ! R is convex, then for any random vector x for which E kxk < 1 and E jg (x)j < 1;

g(E(x))

E (g (x)) :

(B.12)

Conditional Jensen’s Inequality. If g( ) : Rm ! R is convex, then for any random vectors (y; x) for which E kyk < 1 and E kg (y)k < 1; g(E(y j x))

E (g (y) j x) :

Conditional Expectation Inequality. For any r E jE(y j x)jr

(B.13)

such that E jyjr < 1; then

E jyjr < 1:

(B.14)

Expectation Inequality. For any random matrix Y for which E kY k < 1; kE(Y )k

E kY k :

Hölder’s Inequality. If p > 1 and q > 1 and p1 + 1q = 1; then for any random m and Y; (E kXkp )1=p (E kY kq )1=q : E X 0Y

(B.15) n matrices X (B.16)

320

APPENDIX B. PROBABILITY Cauchy-Schwarz Inequality. For any random m E kXk2

E X 0Y

n matrices X and Y; 1=2

E kY k2

1=2

:

(B.17)

Matrix Cauchy-Schwarz Inequality. Tripathi (1999). For any random x 2 Rm and y 2 R` , Eyx0 Exx0 Minkowski’s Inequality. For any random m

Exy 0

Eyy 0

(B.18)

n matrices X and Y;

(E kXkp )1=p + (E kY kp )1=p

(E kX + Y kp )1=p Liapunov’s Inequality. For any random m

(E kXkr )1=r

n matrix X and 1

r

(B.19) p;

(E kXkp )1=p

(B.20)

Markov’s Inequality (standard form). For any random vector x and non-negative function g(x) 0; 1 Pr(g(x) > ) Eg(x): (B.21) Markov’s Inequality (strong form). For any random vector x and non-negative function g(x) 0; 1 Pr(g(x) > ) E (g (x) 1 (g(x) > )) : (B.22) Chebyshev’s Inequality. For any random variable x; Pr(jx

Exj > )

var (x) 2

:

(B.23)

Proof of Jensen’s Inequality (B.12). Since g(u) is convex, at any point u there is a nonempty set of subderivatives (linear surfaces touching g(u) at u but lying below g(u) for all u). Let a + b0 u be a subderivative of g(u) at u = Ex: Then for all u; g(u) a + b0 u yet g(Ex) = a + b0 Ex: Applying expectations, Eg(x) a + b0 Ex = g(Ex); as stated. Proof of Conditional Jensen’s Inequality. The same as the proof of (B.12), but using conditional expectations. The conditional expectations exist since E kyk < 1 and E kg (y)k < 1: Proof of Conditional Expectation Inequality. As the function jujr is convex for r Conditional Jensen’s inequality implies jE(y j x)jr

E (jyjr j x) :

Taking unconditional expectations and the law of iterated expectations, we obtain E jE(y j x)jr

EE (jyjr j x) = E jyjr < 1

as required. Proof of Expectation Inequality. By the Triangle inequality, for k U 1 + (1

)U 2 k

kU 1 k + (1

2 [0; 1];

) kU 2 k

1, the

321

APPENDIX B. PROBABILITY

which shows that the matrix norm g(U ) = kU k is convex. Applying Jensen’s Inequality (B.12) we …nd (B.15). 1 p

Proof of Hölder’s Inequality. Since shows that for any real a and b exp

1 q

+

= 1 an application of Jensen’s Inequality (A.17)

1 1 a+ b p q

1 1 exp (a) + exp (b) : p q

Setting u = exp (a) and v = exp (b) this implies u v + p q

u1=p v 1=q

and this inequality holds for any u > 0 and v > 0: Set u = kXkp =E kXkp and v = kY kq =E kY kq : Note that Eu = Ev = 1: By the matrix Schwarz Inequality (A.11), kX 0 Y k kXk kY k. Thus E kX 0 Y k

p 1=p

(E kXk )

E (kXk kY k)

q 1=q

(E kXkp )1=p (E kY kq )1=q

(E kY k )

= E u1=p v 1=q u v + p q 1 1 = + p q = 1; E

which is (B.16). Proof of Cauchy-Schwarz Inequality. Special case of Hölder’s with p = q = 2: (Eyx0 ) (Exx0 ) x: Note that

Proof of Matrix Cauchy-Schwarz Inequality. De…ne e = y Eee0 0 is positive semi-de…nite. We can calculate that Eee0 = Eyy 0

Eyx0

Exx0

Exy 0 :

Since the left-hand-side is positive semi-de…nite, so is the right-hand-side, which means Eyy 0 (Eyx0 ) (Exx0 ) Exy 0 as stated. Proof of Liapunov’s Inequality. The function g(u) = up=r is convex for u > 0 since p u = kXkr : By Jensen’s inequality, g (Eu) Eg (u) or (E kXkr )p=r

r: Set

E (kXkr )p=r = E kXkp :

Raising both sides to the power 1=p yields (E kXkr )1=r

(E kXkp )1=p as claimed.

Proof of Minkowski’s Inequality. Note that by rewriting, using the triangle inequality (A.12), and then Hölder’s Inequality to the two expectations E kX + Y kp = E kX + Y k kX + Y kp E kXk kX + Y kp

1

1

+ E kY k kX + Y kp

(E kXkp )1=p E kX + Y kq(p =

1)

1=q

1

+ (E kY kp )1=p E kX + Y kq(p

(E kXkp )1=p + (E kY kp )1=p E (kX + Y kp )(p

1)=p

1)

1=q

322

APPENDIX B. PROBABILITY

where the second equality picks q to satisfy 1=p+1=q = 1; and the …nal equality uses this fact to make the substitution q = p=(p 1) and then collects terms. Dividing both sides by E (kX + Y kp )(p 1)=p ; we obtain (B.19). Proof of Markov’s Inequality. Let F denote the distribution function of x: Then Z Pr (g(x) ) = dF (u) fg(u)

g

fg(u)

g

Z

=

1

=

1

Z

g(u)

dF (u)

1 (g(u) > ) g(u)dF (u)

E (g (x) 1 (g(x) > ))

the inequality using the region of integration fg(u) > g: This establishes the strong form (B.22). 1 E (g(x)) ; establishing the standard Since 1 (g(x) > ) 1; the …nal expression is less than form (B.21). Proof of Chebyshev’s Inequality. De…ne y = (x Ex)2 and note that Ey = var (x) : The events fjx Exj > g and y > 2 are equal, so by an application Markov’s inequality we …nd Pr(jx

Exj > ) = Pr(y >

2

2

)

E (y) =

2

var (x)

as stated.

B.11

Maximum Likelihood

In this section we provide a brief review of the asymptotic theory of maximum likelihood estimation. When the density of y i is f (y j ) where F is a known distribution function and 2 is an unknown m 1 vector, we say that the distribution is parametric and that is the parameter of the distribution F: The space is the set of permissible value for : In this setting the method of maximum likelihood is an appropriate technique for estimation and inference on : We let denote a generic value of the parameter and let 0 denote its true value. The joint density of a random sample (y 1 ; :::; y n ) is fn (y 1 ; :::; y n j ) =

n Y i=1

f (y i j ) :

The likelihood of the sample is this joint density evaluated at the observed sample values, viewed as a function of . The log-likelihood function is its natural logarithm log L( ) =

n X i=1

log f (y i j ) :

The likelihood score is the derivative of the log-likelihood, evaluated at the true parameter value. @ Si = log f (y i j 0 ) : @ We also de…ne the Hessian @2 H= E log f (y i j 0 ) (B.24) @ @ 0

323

APPENDIX B. PROBABILITY and the outer product matrix = E S i S 0i :

(B.25)

We now present three important features of the likelihood.

Theorem B.11.1 @ E log f (y j ) @

=0 =

(B.26)

0

ES i = 0

(B.27)

and H=

I

(B.28)

The matrix I is called the information, and the equality (B.28) is called the information matrix equality. The maximum likelihood estimator (MLE) ^ is the parameter value which maximizes the likelihood (equivalently, which maximizes the log-likelihood). We can write this as ^ = argmax log L( ):

(B.29)

2

In some simple cases, we can …nd an explicit expression for ^ as a function of the data, but these cases are rare. More typically, the MLE ^ must be found by numerical methods. To understand why the MLE ^ is a natural estimator for the parameter observe that the standardized log-likelihood is a sample average and an estimator of E log f (y i j ) : n

1X 1 p log L( ) = log f (y i j ) ! E log f (y i j ) : n n i=1

As the MLE ^ maximizes the left-hand-side, we can see that it is an estimator of the maximizer of the right-hand-side. The …rst-order condition for the latter problem is 0=

@ E log f (y i j ) @

which holds at = 0 by (B.26). This suggests that ^ is an estimator of 0 : In. fact, under p conventional regularity conditions, ^ is consistent, ^ ! 0 as n ! 1: Furthermore, we can derive its asymptotic distribution.

Theorem B.11.2 Under regularity conditions,

p

n ^

0

d

! N 0; I

1

.

We omit the regularity conditions for Theorem B.11.2, but the result holds quite broadly for models which are smooth functions of the parameters. Theorem B.11.2 gives the general form for the asymptotic distribution of the MLE. A famous result shows that the asymptotic variance is the smallest possible.

324

APPENDIX B. PROBABILITY Theorem B.11.3 Cramer-Rao Lower Bound. If e is an unbiased regular estimator of ; then var(e) (nI) :

The Cramer-Rao Theorem shows that the …nite sample variance of an unbiased estimator is bounded below by (nI) 1 : This means that the asymptotic variance of the standardized estimator p e 1 n : In other words, the best possible asymptotic variance among 0 is bounded below by I

all (regular) estimators is I 1 : An estimator is called asymptotically e¢cient if its asymptotic variance equals this lower bound. Theorem B.11.2 shows that the MLE has this asymptotic variance, and is thus asymptotically e¢cient.

Theorem B.11.4 The MLE is asymptotically e¢cient in the sense that its asymptotic variance equals the Cramer-Rao Lower Bound.

Theorem B.11.4 gives a strong endorsement for the MLE in parametric models. Finally, consider functions of parameters. If = g( ) then the MLE of is b = g(b): This is because maximization (e.g. (B.29)) is una¤ected by parameterization and transformation. Applying the Delta Method to Theorem B.11.2 we conclude that p

p ' G0 n b

n b

d

! N 0; G0 I

1

(B.30)

G

where G = @@ g( 0 )0 : By Theorem B.11.4, b is an asymptotically e¢cient estimator for since it is the MLE. The asymptotic variance G0 I 1 G is the Cramer-Rao lower bound for estimation of . Theorem B.11.5 The Cramer-Rao lower bound for and the MLE b = g(b) is asymptotically e¢cient.

= g( ) is G0 I

Proof of Theorem B.11.1. To see (B.26); @ E log f (y j ) @

@ @ Z

= =

0

Z

log f (y j ) f (y j

f (y j 0 ) @ f (y j ) dy @ f (y j ) Z @ f (y j ) dy @ = 0

= =

@ 1 @

=

= 0: =

0

Equation (B.27) follows by exchanging integration and di¤erentiation E

@ log f (y j @

0 ) dy

0)

=

@ E log f (y j @

0)

= 0:

=

=

0

0

1

G,

325

APPENDIX B. PROBABILITY Similarly, we can show that @2 @ @

f (y j 0 ) f (y j 0 )

E

0

!

= 0:

By direction computation, @2 @ @

0

log f (y j

@2 @ @

0) =

f (y j 0 ) f (y j 0 )

@ @

f (y j 0 ) f (y j 0 )

@ log f (y j @

0

@2 @ @

=

@ 0) @

f (y j

0 0)

2 0)

f (y j

0

f (y j

0)

@ log f (y j @

0 0) :

Taking expectations yields (B.28). Proof of Theorem B.11.2 Taking the …rst-order condition for maximization of log L( ), and making a …rst-order Taylor series expansion, @ log L( ) @ =^ n X @ log f y i j ^ = @

0 =

i=1 n X

=

i=1

@ log f (y i j @

0) +

n X i=1

@2 @ @

0

log f (y i j

^

n)

0

;

where n lies on a line segment joining ^ and 0 : (Technically, the speci…c value of row in this expansion.) Rewriting this equation, we …nd ^

0

n X

=

i=1

@2 @ @

0

!

log f (y i j

1

n X

n)

Si

i=1

n

varies by

!

where S i are the likelihood scores. Since the score S i is mean-zero (B.27) with covariance matrix (equation B.25) an application of the CLT yields n

1 X d p S i ! N (0; ) : n i=1

The analysis of the sample Hessian is somewhat more complicated due to the presence of n : 2 p Let H( ) = @ @@ 0 log f (y i ; ) : If it is continuous in ; then since n ! 0 it follows that H(

n)

p

! H and so n

1 X @2 n @ @ i=1

n

0

log f (y i ;

n)

=

1X n i=1

p

@2 @ @

0

log f (y i ;

n)

H(

n)

+ H(

n)

!H

by an application of a uniform WLLN. (By uniform, we mean that the WLLN holds uniformly over the parameter value. This requires the second derivative to be a smooth function of the parameter.) Together, p

n ^

0

d

!H

1

N (0; ) = N 0; H

the …nal equality using Theorem B.11.1 .

1

H

1

= N 0; I

1

;

326

APPENDIX B. PROBABILITY Proof of Theorem B.11.3. Let Y = (y 1 ; :::; y n ) be the sample, and set @ log fn (Y ; @

S=

0) =

n X

Si

i=1

which by Theorem (B.11.1) has mean zero and variance nI: Write the estimator e = e (Y ) as a function of the data. Since e is unbiased for any ; Z e = E = e (Y ) f (Y ; ) dY :

Di¤erentiating with respect to and evaluating at 0 yields Z Z @ @ e I= (Y ) 0 f (Y ; ) dY = e (Y ) 0 log f (Y ; ) f (Y ; @ @ the …nal equality since E (S) = 0 By the matrix Cauchy-Schwarz inequality (B.18), E nI; var e

= E E

e

e

= E SS 0 = (nI) as stated.

0

0

e

0

e

0 ) dY

0

= E eS 0 = E

0

S0

S 0 = I; and var (S) = E (SS 0 ) =

0

S 0 E SS 0

e

E S e

0

0

Appendix C

Numerical Optimization Many econometric estimators are de…ned by an optimization problem of the form ^ = argmin Q( )

(C.1)

2

where the parameter is 2 Rm and the criterion function is Q( ) : ! R: For example NLLS, GLS, MLE and GMM estimators take this form. In most cases, Q( ) can be computed for given ; but ^ is not available in closed form. In this case, numerical methods are required to obtain ^:

C.1

Grid Search

Many optimization problems are either one dimensional (m = 1) or involve one-dimensional optimization as a sub-problem (for example, a line search). In this context grid search may be employed. Grid Search. Let = [a; b] be an interval. Pick some " > 0 and set G = (b a)=" to be the number of gridpoints. Construct an equally spaced grid on the region [a; b] with G gridpoints, which is { (j) = a + j(b a)=G : j = 0; :::; Gg. At each point evaluate the criterion function and …nd the gridpoint which yields the smallest value of the criterion, which is (^ |) where |^ = ^ argmin0 j G Q( (j)): This value (^ |) is the gridpoint estimate of : If the grid is su¢ciently …ne to capture small oscillations in Q( ); the approximation error is bounded by "; that is, (^ |) ^ ": Plots of Q( (j)) against (j) can help diagnose errors in grid selection. This method is quite robust but potentially costly: Two-Step Grid Search. The gridsearch method can be re…ned by a two-step execution. For an error bound of " pick G so that G2 = (b a)=" For the …rst step de…ne an equally spaced grid on the region [a; b] with G gridpoints, which is { (j) = a + j(b a)=G : j = 0; :::; Gg: At each point evaluate the criterion function and let |^ = argmin0 j G Q( (j)). For the second step de…ne an equally spaced grid on [ (^ | 1); (^ | + 1)] with G gridpoints, which is { 0 (k) = (^ | 1) + 2k(b a)=G2 : k = 0; :::; Gg: Let k^ = argmin0 k G Q( 0 (k)): The estimate of ^ is k^ . The advantage of the two-step method over a one-step grid search is that the number of p function evaluations has been reduced from (b a)=" to 2 (b a)=" which can be substantial. The disadvantage is that if the function Q( ) is irregular, the …rst-step grid may not bracket ^ which thus would be missed.

C.2

Gradient Methods

Gradient Methods are iterative methods which produce a sequence are designed to converge to ^: All require the choice of a starting value 327

i 1;

: i = 1; 2; ::: which and all require the

328

APPENDIX C. NUMERICAL OPTIMIZATION computation of the gradient of Q( ) g( ) =

@ Q( ) @

and some require the Hessian

@2 Q( ): @ @ 0 If the functions g( ) and H( ) are not analytically available, they can be calculated numerically. Take the j 0 th element of g( ): Let j be the j 0 th unit vector (zeros everywhere except for a one in the j 0 th row). Then for " small H( ) =

gj ( ) '

Q( +

j ")

Q( )

"

:

Similarly, Q( + k ") Q( + j ") + Q( ) "2 In many cases, numerical derivatives can work well but can be computationally costly relative to analytic derivatives. In some cases, however, numerical derivatives can be quite unstable. Most gradient methods are a variant of Newton’s method which is based on a quadratic approximation. By a Taylor’s expansion for close to ^ gjk ( ) '

Q( +

j"

+

k ")

0 = g(^) ' g( ) + H( ) ^ which implies This suggests the iteration rule

^= ^i+1 =

H( ) i

1

H( i )

g( ): 1

g( i ):

where One problem with Newton’s method is that it will send the iterations in the wrong direction if H( i ) is not positive de…nite. One modi…cation to prevent this possibility is quadratic hill-climbing which sets ^i+1 = i (H( i ) + i I m ) 1 g( i ): where i is set just above the smallest eigenvalue of H( i ) if H( ) is not positive de…nite. Another productive modi…cation is to add a scalar steplength i : In this case the iteration rule takes the form Di gi i (C.2) i+1 = i where g i = g( i ) and D i = H( i ) 1 for Newton’s method and Di = (H( i ) + i I m ) 1 for quadratic hill-climbing. Allowing the steplength to be a free parameter allows for a line search, a one-dimensional optimization. To pick i write the criterion function as a function of Q( ) = Q(

i

+ Di gi )

a one-dimensional optimization problem. There are two common methods to perform a line search. A quadratic approximation evaluates the …rst and second derivatives of Q( ) with respect to ; and picks i as the value minimizing this approximation. The half-step method considers the sequence = 1; 1/2, 1/4, 1/8, ... . Each value in the sequence is considered and the criterion Q( i + D i g i ) evaluated. If the criterion has improved over Q( i ), use this value, otherwise move to the next element in the sequence.

329

APPENDIX C. NUMERICAL OPTIMIZATION

Newton’s method does not perform well if Q( ) is irregular, and it can be quite computationally costly if H( ) is not analytically available. These problems have motivated alternative choices for the weight matrix Di : These methods are called Quasi-Newton methods. Two popular methods are do to Davidson-Fletcher-Powell (DFP) and Broyden-Fletcher-Goldfarb-Shanno (BFGS). Let gi = gi =

i

gi

i

1

i 1

and : The DFP method sets Di = Di

1

i 0 i

+

0 i

gi

+

Di

g i g 0i D i g 0i D i 1 g i

1

1

:

The BFGS methods sets Di = Di

1

+

i 0 i

0 i

i

gi

0 i

0 i

gi

2

gi0 D i

1

gi +

i

g 0i D i gi

0 i

1

+

Di

1 0 i

gi gi

0 i

:

For any of the gradient methods, the iterations continue until the sequence has converged in some sense. This can be de…ned by examining whether j i Q ( i 1 )j or jg( i )j i 1 j ; jQ ( i ) has become small.

C.3

Derivative-Free Methods

All gradient methods can be quite poor in locating the global minimum when Q( ) has several local minima. Furthermore, the methods are not well de…ned when Q( ) is non-di¤erentiable. In these cases, alternative optimization methods are required. One example is the simplex method of Nelder-Mead (1965). A more recent innovation is the method of simulated annealing (SA). For a review see Go¤e, Ferrier, and Rodgers (1994). The SA method is a sophisticated random search. Like the gradient methods, it relies on an iterative sequence. At each iteration, a random variable is drawn and added to the current value of the parameter. If the resulting criterion is decreased, this new value is accepted. If the criterion is increased, it may still be accepted depending on the extent of the increase and another randomization. The latter property is needed to keep the algorithm from selecting a local minimum. As the iterations continue, the variance of the random innovations is shrunk. The SA algorithm stops when a large number of iterations is unable to improve the criterion. The SA method has been found to be successful at locating global minima. The downside is that it can take considerable computer time to execute.

Bibliography [1] Abadir, Karim M. and Jan R. Magnus (2005): Matrix Algebra, Cambridge University Press. [2] Aitken, A.C. (1935): “On least squares and linear combinations of observations,” Proceedings of the Royal Statistical Society, 55, 42-48. [3] Akaike, H. (1973): “Information theory and an extension of the maximum likelihood principle.” In B. Petroc and F. Csake, eds., Second International Symposium on Information Theory. [4] Anderson, T.W. and H. Rubin (1949): “Estimation of the parameters of a single equation in a complete system of stochastic equations,” The Annals of Mathematical Statistics, 20, 46-63. [5] Andrews, Donald W. K. (1988): “Laws of large numbers for dependent non-identically distributed random variables,’ Econometric Theory, 4, 458-467. [6] Andrews, Donald W. K. (1991), “Asymptotic normality of series estimators for nonparameric and semiparametric regression models,” Econometrica, 59, 307-345. [7] Andrews, Donald W. K. (1993), “Tests for parameter instability and structural change with unknown change point,” Econometrica, 61, 821-8516. [8] Andrews, Donald W. K. and Moshe Buchinsky: (2000): “A three-step method for choosing the number of bootstrap replications,” Econometrica, 68, 23-51. [9] Andrews, Donald W. K. and Werner Ploberger (1994): “Optimal tests when a nuisance parameter is present only under the alternative,” Econometrica, 62, 1383-1414. [10] Ash, Robert B. (1972): Real Analysis and Probability, Academic Press. [11] Basmann, R. L. (1957): “A generalized classical method of linear estimation of coe¢cients in a structural equation,” Econometrica, 25, 77-83. [12] Bekker, P.A. (1994): “Alternative approximations to the distributions of instrumental variable estimators, Econometrica, 62, 657-681. [13] Billingsley, Patrick (1968): Convergence of Probability Measures. New York: Wiley. [14] Billingsley, Patrick (1995): Probability and Measure, 3rd Edition, New York: Wiley. [15] Bose, A. (1988): “Edgeworth correction by bootstrap in autoregressions,” Annals of Statistics, 16, 1709-1722. [16] Breusch, T.S. and A.R. Pagan (1979): “The Lagrange multiplier test and its application to model speci…cation in econometrics,” Review of Economic Studies, 47, 239-253. [17] Brown, B. W. and Whitney K. Newey (2002): “GMM, e¢cient bootstrapping, and improved inference ,” Journal of Business and Economic Statistics. 330

BIBLIOGRAPHY

331

[18] Carlstein, E. (1986): “The use of subseries methods for estimating the variance of a general statistic from a stationary time series,” Annals of Statistics, 14, 1171-1179. [19] Casella, George and Roger L. Berger (2002): Statistical Inference, 2nd Edition, Duxbury Press. [20] Chamberlain, Gary (1987): “Asymptotic e¢ciency in estimation with conditional moment restrictions,” Journal of Econometrics, 34, 305-334. [21] Choi, In and Peter C.B. Phillips (1992): “Asymptotic and …nite sample distribution theory for IV estimators and tests in partially identi…ed structural equations,” Journal of Econometrics, 51, 113-150. [22] Chow, G.C. (1960): “Tests of equality between sets of coe¢cients in two linear regressions,” Econometrica, 28, 591-603. [23] Cragg, John (1992): “Quasi-Aitken Estimation for Heterskedasticity of Unknown Form" Journal of Econometrics, 54, 179-201. [24] Davidson, James (1994): Stochastic Limit Theory: An Introduction for Econometricians. Oxford: Oxford University Press. [25] Davison, A.C. and D.V. Hinkley (1997): Bootstrap Methods and their Application. Cambridge University Press. [26] Dickey, D.A. and W.A. Fuller (1979): “Distribution of the estimators for autoregressive time series with a unit root,” Journal of the American Statistical Association, 74, 427-431. [27] Donald Stephen G. and Whitney K. Newey (2001): “Choosing the number of instruments,” Econometrica, 69, 1161-1191. [28] Dufour, J.M. (1997): “Some impossibility theorems in econometrics with applications to structural and dynamic models,” Econometrica, 65, 1365-1387. [29] Efron, Bradley (1979): “Bootstrap methods: Another look at the jackknife,” Annals of Statistics, 7, 1-26. [30] Efron, Bradley (1982): The Jackknife, the Bootstrap, and Other Resampling Plans. Society for Industrial and Applied Mathematics. [31] Efron, Bradley and R.J. Tibshirani (1993): An Introduction to the Bootstrap, New York: Chapman-Hall. [32] Eicker, F. (1963): “Asymptotic normality and consistency of the least squares estimators for families of linear regressions,” Annals of Mathematical Statistics, 34, 447-456. [33] Engle, Robert F. and Clive W. J. Granger (1987): “Co-integration and error correction: Representation, estimation and testing,” Econometrica, 55, 251-276. [34] Frisch, Ragnar (1933): “Editorial,” Econometrica, 1, 1-4. [35] Frisch, Ragnar and F. Waugh (1933): “Partial time regressions as compared with individual trends,” Econometrica, 1, 387-401. [36] Gallant, A. Ronald and D.W. Nychka (1987): “Seminonparametric maximum likelihood estimation,” Econometrica, 55, 363-390. [37] Gallant, A. Ronald and Halbert White (1988): A Uni…ed Theory of Estimation and Inference for Nonlinear Dynamic Models. New York: Basil Blackwell.

BIBLIOGRAPHY

332

[38] Galton, Francis (1886): “Regression Towards Mediocrity in Hereditary Stature,” The Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246-263. [39] Goldberger, Arthur S. (1991): A Course in Econometrics. Cambridge: Harvard University Press. [40] Go¤e, W.L., G.D. Ferrier and J. Rogers (1994): “Global optimization of statistical functions with simulated annealing,” Journal of Econometrics, 60, 65-99. [41] Gauss, K.F. (1809): “Theoria motus corporum coelestium,” in Werke, Vol. VII, 240-254. [42] Granger, Clive W. J. (1969): “Investigating causal relations by econometric models and cross-spectral methods,” Econometrica, 37, 424-438. [43] Granger, Clive W. J. (1981): “Some properties of time series data and their use in econometric speci…cation,” Journal of Econometrics, 16, 121-130. [44] Granger, Clive W. J. and Timo Teräsvirta (1993): Modelling Nonlinear Economic Relationships, Oxford University Press, Oxford. [45] Gregory, A. and M. Veall (1985): “On formulating Wald tests of nonlinear restrictions,” Econometrica, 53, 1465-1468, [46] Haavelmo, T. (1944): “The probability approach in econometrics,” Econometrica, supplement, 12. [47] Hall, A. R. (2000): “Covariance matrix estimation and the power of the overidentifying restrictions test,” Econometrica, 68, 1517-1527, [48] Hall, P. (1992): The Bootstrap and Edgeworth Expansion, New York: Springer-Verlag. [49] Hall, P. (1994): “Methodology and theory for the bootstrap,” Handbook of Econometrics, Vol. IV, eds. R.F. Engle and D.L. McFadden. New York: Elsevier Science. [50] Hall, P. and J.L. Horowitz (1996): “Bootstrap critical values for tests based on GeneralizedMethod-of-Moments estimation,” Econometrica, 64, 891-916. [51] Hahn, J. (1996): “A note on bootstrapping generalized method of moments estimators,” Econometric Theory, 12, 187-197. [52] Hamilton, James D. (1994) Time Series Analysis. [53] Hansen, Bruce E. (1992): “E¢cient estimation and testing of cointegrating vectors in the presence of deterministic trends,” Journal of Econometrics, 53, 87-121. [54] Hansen, Bruce E. (1996): “Inference when a nuisance parameter is not identi…ed under the null hypothesis,” Econometrica, 64, 413-430. [55] Hansen, Bruce E. (2006): “Edgeworth expansions for the Wald and GMM statistics for nonlinear restrictions,” Econometric Theory and Practice: Frontiers of Analysis and Applied Research, edited by Dean Corbae, Steven N. Durlauf and Bruce E. Hansen. Cambridge University Press. [56] Hansen, Lars Peter (1982): “Large sample properties of generalized method of moments estimators, Econometrica, 50, 1029-1054. [57] Hansen, Lars Peter, John Heaton, and A. Yaron (1996): “Finite sample properties of some alternative GMM estimators,” Journal of Business and Economic Statistics, 14, 262-280.

BIBLIOGRAPHY

333

[58] Hausman, J.A. (1978): “Speci…cation tests in econometrics,” Econometrica, 46, 1251-1271. [59] Heckman, J. (1979): “Sample selection bias as a speci…cation error,” Econometrica, 47, 153161. [60] Horowitz, Joel (2001): “The Bootstrap,” Handbook of Econometrics, Vol. 5, J.J. Heckman and E.E. Leamer, eds., Elsevier Science, 3159-3228. [61] Imbens, G.W. (1997): “One step estimators for over-identi…ed generalized method of moments models,” Review of Economic Studies, 64, 359-383. [62] Imbens, G.W., R.H. Spady and P. Johnson (1998): “Information theoretic approaches to inference in moment condition models,” Econometrica, 66, 333-357. [63] Jarque, C.M. and A.K. Bera (1980): “E¢cient tests for normality, homoskedasticity and serial independence of regression residuals, Economic Letters, 6, 255-259. [64] Johansen, S. (1988): “Statistical analysis of cointegrating vectors,” Journal of Economic Dynamics and Control, 12, 231-254. [65] Johansen, S. (1991): “Estimation and hypothesis testing of cointegration vectors in the presence of linear trend,” Econometrica, 59, 1551-1580. [66] Johansen, S. (1995): Likelihood-Based Inference in Cointegrated Vector Auto-Regressive Models, Oxford University Press. [67] Johansen, S. and K. Juselius (1992): “Testing structural hypotheses in a multivariate cointegration analysis of the PPP and the UIP for the UK,” Journal of Econometrics, 53, 211-244. [68] Kitamura, Y. (2001): “Asymptotic optimality and empirical likelihood for testing moment restrictions,” Econometrica, 69, 1661-1672. [69] Kitamura, Y. and M. Stutzer (1997): “An information-theoretic alternative to generalized method of moments,” Econometrica, 65, 861-874.. [70] Koenker, Roger (2005): Quantile Regression. Cambridge University Press. [71] Kunsch, H.R. (1989): “The jackknife and the bootstrap for general stationary observations,” Annals of Statistics, 17, 1217-1241. [72] Kwiatkowski, D., P.C.B. Phillips, P. Schmidt, and Y. Shin (1992): “Testing the null hypothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root?” Journal of Econometrics, 54, 159-178. [73] Lafontaine, F. and K.J. White (1986): “Obtaining any Wald statistic you want,” Economics Letters, 21, 35-40. [74] Lehmann, E.L. and George Casella (1998): Theory of Point Estimation, 2nd Edition, Springer. [75] Lehmann, E.L. and Joseph P. Romano (2005): Testing Statistical Hypotheses, 3rd Edition, Springer. [76] Li, Qi and Je¤rey Racine (2007) Nonparametric Econometrics. [77] Lovell, M.C. (1963): “Seasonal adjustment of economic time series,” Journal of the American Statistical Association, 58, 993-1010.

BIBLIOGRAPHY

334

[78] MacKinnon, James G. (1990): “Critical values for cointegration,” in Engle, R.F. and C.W. Granger (eds.) Long-Run Economic Relationships: Readings in Cointegration, Oxford, Oxford University Press. [79] MacKinnon, James G. and Halbert White (1985): “Some heteroskedasticity-consistent covariance matrix estimators with improved …nite sample properties,” Journal of Econometrics, 29, 305-325. [80] Magnus, J. R., and H. Neudecker (1988): Matrix Di¤erential Calculus with Applications in Statistics and Econometrics, New York: John Wiley and Sons. [81] Mann, H.B. and A. Wald (1943). “On stochastic limit and order relationships,” The Annals of Mathematical Statistics 14, 217–226. [82] Muirhead, R.J. (1982): Aspects of Multivariate Statistical Theory. New York: Wiley. [83] Nelder, J. and R. Mead (1965): “A simplex method for function minimization,” Computer Journal, 7, 308-313. [84] Newey, Whitney K. (1990): “Semiparametric e¢ciency bounds,” Journal of Applied Econometrics, 5, 99-135. [85] Newey, Whitney K. and Daniel L. McFadden (1994): “Large Sample Estimation and Hypothesis Testing,” in Robert Engle and Daniel McFadden, (eds.) Handbook of Econometrics, vol. IV, 2111-2245, North Holland: Amsterdam. [86] Newey, Whitney K. and Kenneth D. West (1987): “Hypothesis testing with e¢cient method of moments estimation,” International Economic Review, 28, 777-787. [87] Owen, Art B. (1988): “Empirical likelihood ratio con…dence intervals for a single functional,” Biometrika, 75, 237-249. [88] Owen, Art B. (2001): Empirical Likelihood. New York: Chapman & Hall. [89] Park, Joon Y. and Peter C. B. Phillips (1988): “On the formulation of Wald tests of nonlinear restrictions,” Econometrica, 56, 1065-1083, [90] Phillips, Peter C.B. (1989): “Partially identi…ed econometric models,” Econometric Theory, 5, 181-240. [91] Phillips, Peter C.B. and Sam Ouliaris (1990): “Asymptotic properties of residual based tests for cointegration,” Econometrica, 58, 165-193. [92] Politis, D.N. and J.P. Romano (1996): “The stationary bootstrap,” Journal of the American Statistical Association, 89, 1303-1313. [93] Potscher, B.M. (1991): “E¤ects of model selection on inference,” Econometric Theory, 7, 163-185. [94] Qin, J. and J. Lawless (1994): “Empirical likelihood and general estimating equations,” The Annals of Statistics, 22, 300-325. [95] Ramsey, J. B. (1969): “Tests for speci…cation errors in classical linear least-squares regression analysis,” Journal of the Royal Statistical Society, Series B, 31, 350-371. [96] Rudin, W. (1987): Real and Complex Analysis, 3rd edition. New York: McGraw-Hill. [97] Said, S.E. and D.A. Dickey (1984): “Testing for unit roots in autoregressive-moving average models of unknown order,” Biometrika, 71, 599-608.

BIBLIOGRAPHY

335

[98] Shao, J. and D. Tu (1995): The Jackknife and Bootstrap. NY: Springer. [99] Sargan, J.D. (1958): “The estimation of economic relationships using instrumental variables,” Econometrica, 2 6, 393-415. [100] Shao, Jun (2003): Mathematical Statistics, 2nd edition, Springer. [101] Sheather, S.J. and M.C. Jones (1991): “A reliable data-based bandwidth selection method for kernel density estimation, Journal of the Royal Statistical Society, Series B, 53, 683-690. [102] Shin, Y. (1994): “A residual-based test of the null of cointegration against the alternative of no cointegration,” Econometric Theory, 10, 91-115. [103] Silverman, B.W. (1986): Density Estimation for Statistics and Data Analysis. London: Chapman and Hall. [104] Sims, C.A. (1972): “Money, income and causality,” American Economic Review, 62, 540-552. [105] Sims, C.A. (1980): “Macroeconomics and reality,” Econometrica, 48, 1-48. [106] Staiger, D. and James H. Stock (1997): “Instrumental variables regression with weak instruments,” Econometrica, 65, 557-586. [107] Stock, James H. (1987): “Asymptotic properties of least squares estimators of cointegrating vectors,” Econometrica, 55, 1035-1056. [108] Stock, James H. (1991): “Con…dence intervals for the largest autoregressive root in U.S. macroeconomic time series,” Journal of Monetary Economics, 28, 435-460. [109] Stock, James H. and Jonathan H. Wright (2000): “GMM with weak identi…cation,” Econometrica, 68, 1055-1096. [110] Theil, H. (1953): “Repeated least squares applied to complete equation systems,” The Hague, Central Planning Bureau, mimeo. [111] Theil, H. (1971): Principles of Econometrics, New York: Wiley. [112] Tobin, James (1958): “Estimation of relationships for limited dependent variables,” Econometrica, 2 6, 24-36. [113] Tripathi, Gautam (1999): “A matrix extension of the Cauchy-Schwarz inequality,” Economics Letters, 63, 1-3. [114] van der Vaart, A.W. (1998): Asymptotic Statistics, Cambridge University Press. [115] Wald, A. (1943): “Tests of statistical hypotheses concerning several parameters when the number of observations is large,” Transactions of the American Mathematical Society, 54, 426-482. [116] Wang, J. and E. Zivot (1998): “Inference on structural parameters in instrumental variables regression with weak instruments,” Econometrica, 66, 1389-1404. [117] White, Halbert (1980): “A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity,” Econometrica, 48, 817-838. [118] White, Halbert (1984): Asymptotic Theory for Econometricians, Academic Press. [119] Wooldridge, Je¤rey M. (2002) Econometric Analysis of Cross Section and Panel Data, MIT Press.

BIBLIOGRAPHY

336

[120] Zellner, Arnold. (1962): “An e¢cient method of estimating seemingly unrelated regressions, and tests for aggregation bias,” Journal of the American Statistical Association, 57, 348-368. [121] Zhang, Fuzhen and Qingling Zhang (2006): “Eigenvalue inequalities for matrix product,” IEEE Transactions on Automatic Control, 51, 1506-1509.