Lectures on some aspects of the bootstrap

Citation preview

1

´ ´ e de Calcul de Probabilit´ Ecole d’Et´ es de Saint–Flour, 1996

LECTURES ON SOME ASPECTS OF THE BOOTSTRAP

´∗ by Evarist Gine The University of Connecticut

∗

Work partially supported by NSF Grants Nos. DMS-9300725 and DMS-9625457 and by the CNRS through the Laboratoire de Math´ ematiques Appliqu´ ees de l’Universit´ e Blaise Pascal, Clermont–Ferrand, France.

2 Table of Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Chapter 1: On the bootstrap in R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1. Efron’s bootstrap of the mean in R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1.1. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.1.2. About the proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2. The general exchangeable bootstrap of the mean . . . . . . . . . . . . . . . . . . . 18 1.3. The bootstrap of the mean for stationary sequences . . . . . . . . . . . . . . . 23 1.4. The bootstrap of U–statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 1.5. A general m out of n bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Chapter 2: On the bootstrap for empirical processes . . . . . . . . . . . . . . . . . . . . . . 47 2.1. Background from empirical process theory . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.1.1. Convergence in law of sample bounded processes . . . . . . . . . . . . . 47 2.1.2. Symmetrization, L´evy type and Hoffmann–Jørgensen inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 2.1.3. Donsker classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 2.2. Poissonization inequalities and Efron’s bootstrap in probability . . . . 67 2.3. The almost sure Efron’s bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 2.4. The exchangeable bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 2.5. Uniformly pregaussian classes of functions and the bootstrap . . . . . . 92 2.6. Some remarks on applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 2.6.1. The bootstrap of the median . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 2.6.2. The delta method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 2.6.3. M–estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

3 Preface Let X, Xi , i ∈ N, be i.i.d.(P) and let Hn := Hn (X1 , . . . , Xn ; P) be a root (that is, a function of both the data and their common probability law), symmetric in the x entries, whose law under P we would like to estimate. If Qn approximates P (e.g. in the sense of convergence in distribution) and {Xi∗ }ni=1 are i.i.d.(Qn ), then, under the appropriate hypotheses, the probability law of Hn∗ := Hn (X1∗ , . . . , Xn∗ ; Qn ) may approximate that of Hn . This principle would be useful if the law of Hn∗ were easier to obtain or, at least, a large number of independent samples from the law Qn could be easily produced in order to approximate it. In 1979, in a landmark P paper, Efron proposed (among other things) to take Qn = Pn (ω) with n Pn (ω) = i=1 δXi (ω) /n, the empirical distribution, and gave this procedure the name of bootstrap. This makes very good sense because Pn (ω) →w P a.s. in great generality, and resampling from Pn (ω), in our computer era, is easy. As he pointed out, the empirical distribution is not the only possible candidate for Qn , particularly if a restricted model is assumed. For instance, the variables Xi are i.i.d. √ suppose ¯ n − θ), where X ¯ n is the sample N (θ, σ 2 ), θ and σ 2 unknown, and take Hn =P n(X n ¯ n , s2 ) (s2 = ¯ 2 mean. If we take Qn = N (X n n i=1 (Xi − Xn ) /(n − 1), the sample ∗ variance), then, estimating the law of Hn by that√of Hn , which is N (0, s2n ), amounts ¯ n − θ)/sn , by N (0, 1). In this to estimating the law of the Student t–statistic, n(X sense, the bootstrap has been around for some time in one form or other. Of course, the extraordinary merit of Efron’s proposal consists in the formulation of a basic principle that applies in great generality, both in parametric and in non–parametric settings. In these lectures we propose to study first order consistency of the bootstrap in the simple but important case of the mean, taken in a general sense (including e.g. the Kolmogorov–Smirnov statistic since the distribution function F (x) is the mean of the process IX≤x , x ∈ R). Chapter 1 will be devoted to the bootstrap of the mean in finite dimensions, and it will also include the bootstrap of U –statistics and of very general statistics when the bootstrap sample size is reduced. In Chapter 2 we will consider the bootstrap of empirical processes (the bootstrap of the mean in infinite dimensions). ¯ n −EX, one encounters several features of the Already for the simple statistic X bootstrap that are general. For instance, there are situations when the bootstrap approximation works better than the limit law (we will touch only very briefly on this) although this is not always the case since, in particular, the regular bootstrap of the mean (i.e., mimicking the statistic for the empirical distribution instead of the original) does not work in general, neither a.s. nor in probability. The limits of validity of the bootstrap can be exactly determined in our simple situation. There are ways to modify the regular bootstrap when it does not work, such as reducing the bootstrap sample size. In fact, sampling without replacement m times from the n data, with m/n → 0, works in great generality. In other instances, however, reducing the bootstrap sample size is not the only solution and another appropriate course of action may be to devise more complicated sampling plans that better mimick the original random mechanism; one of the first, very simple, instances of this is the bootstrap of degenerate U –statistics. Another way of describing the regular bootstrap of the mean is that instead of the average of the data, one takes a

4 linear combination of the data with multinomial coefficients; then the question arises as to whether other random coefficients are also appropriate, or even better (like, e.g., in the Bayesian bootstrap); many of these different bootstraps are instances of the ‘exchangeable bootstrap’, and therefore we will examine this bootstrap for the sample mean. The bootstrap of the mean when the observations are α–mixing instead of independent is also studied; the moving blocks bootstrap that applies to this situation is another, more sophisticated, departure from the regular Efron’s bootstrap. All these questions will be treated in Chapter 1, which will conclude with consideration of a bootstrap procedure that applies in great generality, the so called ‘m out of n bootstrap without replacement’. As indicated above, in the second part of these lectures we will study the bootstrap for the empirical process indexed by families of functions as general as possible. We will see that whenever the empirical process satisfies the central limit theorem, the bootstrap works, and conversely (in a sense). This is probably the most general statement that can be made regarding consistency of the bootstrap: both, directly and via the delta method, this result validates the bootstrap for a great wealth of statistics. If one restricts the class of functions, but still remaining within a very general situation, it can also be proved that basically any sensible model based bootstrap works for the empirical process, including the smooth bootstrap, the symmetric bootstrap, the parametric bootstrap, a ‘projection onto the model’ bootstrap, etc. Finally, as an application of the bootstrap for empirical processes, we will consider the bootstrap of M –estimators. I thank my wife Rosalind for her constant support and extraordinary patience during the writing of these notes. I thank Dragan Radulovi´c and Jon Wellner for personal comunications that made their way into these notes and for reading parts ´ ´ e de Saint–Flour for of a first draft. Thanks also to the organizers of the Ecole d’Et´ the opportunity to prepare these lectures and thanks as well to all the participants in the course for their comments, their interest and their patience. I would like to mention here that both, these lectures and the lectures on decoupling and U – statistics in this volume have their origin in a short course on these same topics that I gave at the Universit´e de Paris-Sud (Orsay) in 1993. It is therefore a pleasure for me to also extend my gratitude to the Orsay Statistics group.

Saint–Flour, Storrs, 1996

Evarist Gin´e

5 Chapter 1: On the bootstrap in R In this ¡chapter we¢ study the consistency of the bootstrap mostly for the statistic ¯ Hn = na−1 n Xn −EX . We will see that it is not always possible to bootstrap it, not even in probability, and will explore the limits of validity of the bootstrap procedure. We will also see how it can be made consistent by reducing the bootstrap sample size. This will be done in Section 1. A more general bootstrap (the ‘exchangeable bootstrap’) will be considered in Section 2. Section 3 is devoted to the bootstrap of the mean for mixing observations. The next section is devoted to the bootstrap of U –statistics, as a simple instance of the need to adapt the bootstrap procedure so as to mimic the main features of the original statistic. Finally, we present in Section 5 a bootstrap procedure (the m out of n bootstrap without replacement) which is consistent in very general situations. 1.1. Efron’s bootstrap of the mean in R. In this Section we let X, Xi , i ∈ N, ¯ n = Pn Xi /n, σ 2 = Pn (Xi − X) ¯ 2 /n, be i.i.d.(P), and set X = {Xi }∞ , X n i=1 i=1 i=1 ∗ , i = 1, . . . , n, are defined to n ∈ N. For each n ∈ N, the bootstrap variables Xn,i be conditionally i.i.d. given the sample X, and with conditional law ¯ ª © ∗ 1 Pr Xn,i = Xj ¯X = , j = 1, . . . , n. n As is customary, we denote Pr(·|X) by Pr∗ (·), and so we do with the conditional law (L∗ ) and the conditional expectation (E∗ ) given the sample X. For instance, if Ui , i ∈ N, are i.i.d. uniform on [0, 1], independent of X, then a realization of the bootstrap sample is ∗ Xn,i

=

n X

Xj IUi ∈A(j,n) , i = 1, . . . , n, n ∈ N,

j=1

where A(j, n) = ((j − 1)/n, j/n]. Without loss of generality we can assume the U ’s and the X’s defined as coordinates in a product probability space, the P U ’s depending n 0 ∗ ∗ ¯ /n. only on ω , the X’s only on ω. The bootstrap sample mean is Xn = i=1 Xn,i Whereas the meaning of the limit ´ ³√ ¡ ∗ ∗ ¯ ¯ n Xn − Xn ) →w µ a.s. (1.1) L ³√ ¡ ¢´ ∗ ¯ ¯ n Xn (ω) − Xn (ω) →w µ, where is clear, namely, that for almost every ω, L ω is fixed and the randomness comes from the U ’s, the meaning of ´ ³√ ¡ ¯∗ − X ¯ n ) →w µ in pr. L∗ n X (1.10 ) n ∗

is less clear and we explain it now. Let d©Rbe a distance metrizing convergence in ª law to µ in R. For example, d(ν, µ) = sup f d(µ − ν) : kf k∞ + kf kBL ≤ 1 (with kf kBL = supx6=y |f (y) − f (x)|/|y − x|), which is in fact a countable sup; or dk (ν, µ), defined as the sup of the same integrals but now over over the k times differentiable Pk functions f such that kf k∞ + i=1 kf (i) k∞ ≤ 1. Moreover, if µ has no atoms, d

6 can be taken to be the sup distance between distribution functions. Then, (1.1’) simply means that ³ ¡√ ¡ ¢ ´ ¯∗ − X ¯ n ) , µ → 0 in pr. d L∗ n X (1.2) n This definition does not depend on the distance used since it is in fact equivalent to (1.1) holding along some subsequence of every subsequence. These definitions extend to any bootstrap functionals Hn∗ . 1.1.1. Results. The following theorem asserts that the mean can be bootstrapped a.s. iff EX 2 < ∞. The direct part is due to Bickel and Freedman (1981) and Singh (1981) and the converse to Gin´e and Zinn (1989). 1.1. Theorem. (a) If EX 2 = σ 2 < ∞ then ³ Pn (X ∗ − X ¯n) ´ i=1 ∗ √n,i →w N (0, σ 2 ) a.s. L n

(1.3)

(b) Conversely, if there exist random variables cn (ω), an increasing sequence {an }∞ n=1 of positive numbers tending to infinity, and a random probability measure µ(ω) non– degenerate with postive probability, such that ´ ³ Pn X ∗ (ω) i=1 n,i ∗ − cn (ω) →w µ(ω) a.s., (1.4) L an √ √ then σ 2 := EX 2 < ∞, n/an → c for some c > 0, µ = N (0, cσ 2 ) a.s. and cn (ω) ¯ n (ω)/an can be taken to be cn (ω) = nX Here is the analogue for the bootstrap in probability: 1.2. Theorem. (a) If X is in domain of attraction of the normal law and the ¢ ¡Pthe n → (X − EX)/a N (0, 1), then constants an are such that L i n w i=1 ³ Pn (X ∗ − X ¯n) ´ n,i i=1 →w N (0, 1) in pr. (1.5) L∗ an (b) Conversely, if L

∗

³ Pn

i=1

∗ ´ Xn,i (ω) − cn (ω) →w µ(ω) in pr. an

(1.6)

with µ(ω) non probability, then there is σ > 0 ¡Pndegenerate on a¢ set of positive 2 such that L i=1 (Xi − EX)/an →w N (0, σ ), in particular, X is in the domain of atraction of the normal law with admissible norming constants an , and µ = N (0, σ 2 ) a.s. Part (a) of this theorem was observed by Athreya (1985) and Part (b) by Gin´e and Zinn (1989). These two theorems set limits to the validity of Efron’s bootstrap in the case of the mean.

7 Theorems 1.1 and 1.2 also hold for i.i.d. random vectors in Rd : The Cram´er– Wold’s device (that is, taking linear combinations of the coordinates) reduces the vector case to R. We should also remark that there is convergence of all bootstrap moments in both (1.3) and (1.5), a.s. in one case, in pr. in the other. In fact, under the hypotheses of Theorem 1.2 (a), we have that for all t > 0, n n X o ∗ ¯ n )/an → EetZ in pr., (Xn,i −X E∗ exp t

(1.50 )

i=1

where Z is N (0, 1), and the analogous statement holds a.s. if EX 2 < ∞. (Arcones and Gin´e, 1991; previously, Bickel and Freedman, 1981, had observed that (1.3) holds with convergence of the second bootstrap moments). (1.5’) justifies bootstrap estimation of variances and other functionals of the original distribution. If EX 2 < ∞ then σn2 → VarX a.s. by the law of large numbers, and Theorem 1.1 (a) gives that ³ Pn (X ∗ − X ¯n) ´ n,i i=1 ∗ √ (1.7) →w N (0, 1) a.s. L σn n Likewise, if EX 2 = ∞ but X is in the domain of atraction of the normal law with norming constants an , as in (a) of Theorem 1.2, then Raikov’s theorem (e.g. Gnedenko–Kolmogorov’s book, or a simple standard argument) easily implies that lim

n→∞

n X i=1

Xi2 /a2n = lim

n→∞

n X

¯ n )2 /a2 = 1 in pr. (Xi − X n

i=1

and therefore, Theorem 1.2 (a) shows ³ Pn (X ∗ − X ¯n) ´ n,i i=1 ∗ √ L →w N (0, 1) in pr. σn n

(1.8)

It is an exercise to check that σn in equations (1.7) and (1.8) can be replaced

by

σn∗ .

The two statements above about the studentized bootstrap clt also have converses. Here is the complete statement: 1.3. Theorem. (a) EX 2 < ∞ if and only if the studentized bootstrap clt holds a.s., that is, iff (1.7) holds. (b) X is in the domain of attraction of the normal law iff the studentized bootstrap clt holds in probability, that is, iff (1.8) holds. Part (a) of this theorem was observed by Cs¨org˝ o and Mason (1989) and part (b) by Hall (1990). The exact conditions under which there exist random normings An → ∞ and ¯ ∗ − Bn )/An }∞ converges in law conditionally random centerings Bn such that {(X n n=1 on X, a.s. or in probability, have been determined respectively by Sepanski (1993) and Hall (1990). Besides normal limits, only Poisson limits are possible and then, the relevant side of the tail is slowly varying at infinity (so, in this case, X is not even

8 in the Feller class). We do not discuss the Poisson limit situation, which corresponds to E|X| = ∞ and does not relate to the bootstrap of the mean. We will not discuss either p–stable domains of attraction with p ≤ 1, for the same reason. Regarding domains of attraction the following two results essentially tell the story: to have consistency of the bootstrap in this case, we must reduce the bootstrap sample size. 1.4. Theorem. Suppose that X is in the domain¡of ¢ Pnatraction of a non–degenerate p–stable law µ, 1 < p < 2, concretely, assume L i=1 (Xi − EX)/an →w µ for some constants an % ∞. Let mn % ∞. Then, ¢ ³ Pmn ¡X ∗ − X ¯n ´ n,i i=1 ∗ L (1.9) →w µ in pr. amn if and only if

mn → 0, n

The direct part of this theorem is due to Athreya (1985) and the converse was observed in Arcones and Gin´e, (1989). As with Theorems 1.1 and 1.2, conditional weak convergence in (1.9) can be strengthened to coditional convergence of bootstrap moments, but here only short of the p–th moment, that is, we have ¢ Z ¯ Pmn ¡X ∗ − X ¯ n ¯¯α n,i i=1 ∗¯ E ¯ (1.90 ) ¯ → |x|α dµ(x) in pr., 0 < α < p amn (Arcones and Gin´e, 1991). 1.5. Theorem. Let X be in the domain of¡P atraction of a non–degenerate p–stable ¢ n law µ, 1 < p ≤ 2, concretely, assume L i=1 (Xi − EX)/an →w µ for some constants an % ∞, and let mn % ∞ be a regular sequence in the sense that lim inf n→∞ mn /m2n > 0. Then, ¢ ³ Pmn ¡X ∗ − X ¯n ´ n,i i=1 ∗ (1.10) →w µ a.s. L amn if

mn log log n → 0, n and (1.10) does not hold if lim inf n→∞ (mn log log n)/n > 0.

(1.11)

This theorem is due to Arcones and Gin´e (1989). Self–normalization is also possible in the previous two theorems. Arcones and Gin´e (1991) show that for X in the domain of attraction of a p–stable law, 1 < p ≤ 2, and for mn /n → 0, ¢ · Pn · Pmn ¡ ∗ ¸ ¯ ¸ i=1 Xn,i − Xn ∗ i=1 (Xi − EX) w − lim L = w − lim L ¡P in pr. (1.12) ¡Pmn ∗2 ¢1/2 ¢1/2 n n→∞ n→∞ 2 i=1 Xn,i i=1 Xi Their proof also shows that this holds a.s. if the sequence mn is regular and satisfies (1.11). Deheuvels, Mason and Shorack (1992) have another approach to a result

9 similar to (1.12), but with the norming in the bootstrap quantity depending on the X’s, as in (1.8). They also prove the following theorem for the bootstrap of the maximum of i.i.d. uniform random variables, one of the first examples of failure of Efron’s bootstrap with mn = n (Bickel and Freedman, 1981). Let X be uniform on (0, θ). Then, it is easy to see that ¢´ ³ n¡θ − max 1≤i≤n |Xi | →w µ, (1.13) L θ where µ is the exponential distribution with unit parameter. Here is the Deheuvels et al. bootstrap version of this limit. 1.6. Theorem. Let X and µ be as in (1.13). Then, if mn /n → 0 we have ³ mn ¡max1≤i≤n |Xi | − max1≤i≤m |X ∗ |¢ ´ n n,i →w µ in pr., L∗ max1≤i≤n |Xi |

(1.14)

and if (mn log log n)/n → 0 then the limit in (1.14) holds a.s. ¡Pn ¢ ∗ ¯ n )/σn √n (Xn,i −X Back to Theorem 1.3 (a), one may ask how good is L∗ i=1 ¡Pn √ ¢ as an approximation to L i=1 (Xi − EX)/σ n . This has been thoroughly studied, starting with Singh (1981), who showed that if E|X|3 < ∞ and X is skewed, the bootstrap approximation may be better than the normal approximation. Hall (1988) shows that in case E|X|3 = ∞ the bootstrap approximation can actually do worse. We will present here a weaker and simpler result on direct comparison of the bootstrap and the original distributions which also indicates how the bootstrap improves on the normal approximation for skewed random variables with finite third moment. D. Radulovi´c showed this to me and I thank him for allowing me to report on his arguments in these lectures. For probability measures µ, ν on R, define n¯ Z o ¯ d4 (µ, ν) = sup ¯ f d(µ − ν)¯ : kf k∞ ≤ 1, kf (i) k∞ ≤ 1, 1 ≤ i ≤ 3 , a distance that metrizes weak convergence of probability measures on R. It is easy to see, using a Lindeberg type argument, that if E|X|3 < ∞ and, without loss of generality, EX = 0 and EX 2 = 1, then · ³ Pn ¸ ´ ¢ ¡ i=1 Xi √ d4 L (1.15) , N (0, 1) = O n−1/2 , n and that this cannot in general be improved if EX 3 6= 0. For the bootstrap, we have: 1.7. Proposition. If E|X|3 < ∞ then · ³ Pn ∗ ¯ ´ ³ Pn (Xi − EX) ´¸ ¢ ¡ i=1 (Xn,i − Xn ) ∗ i=1 √ √ ,L = o n−1/2 a.s. d4 L σn n σ n

(1.16)

10 1.1.2. About the proofs. It is worth noting that as long as the norming constants amn ∗ tend to infinity, the triangular array {Xn,i : i = 1, . . . , mn }∞ n=1 is a.s. infinitesimal: ∗

max Pr

i≤mn

©

∗ |Xn,i |

ª

∗

> δamn = Pr

©

∗ |Xn,1 |

> δamn

ª

1X = I|Xj |>δamn , n j=1 n

and, by the law of large numbers, the limsup of this last sum is a.s. bounded by E|X|I|X|>c for all c > 0; letting c → ∞ along a countable sequence gives the a.s. infinitesimality. Then, the proofs of Theorems 1.1 to 1.5 consist in i) showing bootstrap convergence by just checking the conditions for the general normal (or stable) convergence criterion for infinitesimal arrays and ii) applying the converse part of this criterion to infer properties on the distribution of X from bootstrap convergence. This program provides relatively simple proofs, except for Theorem 1.5, where one must proceed, roughly speaking, as in the proof of the LIL. Similar comments apply to Theorem 1.6. (1.12) just follows from the bootstrap of a stable convergence theorem in R2 , and its proof is not different from that of the direct parts of Theorems 1.4 and 1.5. Finally, Proposition 1.7 is proved by a Lindeberg type argument applied to a simple coupling between the bootstrap and the original statistics. We will give complete proofs of Theorem 1.1 and Proposition 1.7, and then indicate parts of proofs of the other results. 1.7. Proof of Theorem 1.1 a). We will prove that if EX 2 < ∞ and if mn → ∞ then ³ Pmn (X ∗ − X ¯n) ´ n,i i=1 ∗ (1.30 ) →w N (0, σ 2 ) a.s. L √ mn By the general criterion for normal convergence (e.g., Araujo and Gin´e, 1980, Cor. 2.4.8, p.63), it suffices to prove © ∗ ª (1.17) | > δmn1/2 → 0 a.s. mn Pr∗ |Xn,1 for all δ > 0,

¡ ∗ I|X ∗ Var∗ Xn,1

¢

1/2 n,1 |≤mn

and

∗ mn1/2 E∗ Xn,1 I|X ∗

→ VarX a.s.

(1.18)

→ 0 a.s.

(1.19)

1/2 n,1 |>mn

(Then, one makes the set of measure 1 where convergence takes place in (1.17) independent of δ just by taking a countable dense set of δ’s, which is all that is needed.) The basic observation is that, since EX 2 < ∞, the law of large numbers yields n 1X |Xj |p I|Xj |>δm1/2 → 0 a.s., 0 < p ≤ 2, δ > 0. (1.20) n n j=1 1/2

(Replace δmn by c rational and take limits first as n → ∞ and then as c → ∞.) Then, (1.17) and (1.19) follow immediately because ∗

mn Pr

©

∗ | |Xn,1

>

δmn1/2

ª

n n δ −2 X 2 mn X I X I = 1/2 ≤ 1/2 n j=1 |Xj |>δmn n j=1 j |Xj |>δmn

11 and

¯

¯

∗ mn1/2 ¯E∗ Xn,1 I|X ∗ |>m1/2 ¯ n n,1

n δ −1 X 2 ≤ X I 1/2 . n j=1 j |Xj |>mn

As for (1.18), using (1.20) and the law of large numbers once more, we obtain: ∗

Var

¡

∗ Xn,1 I|X ∗ |≤m1/2 n n,1

¢

n n i2 h1 X 1X 2 = X I Xj I|Xj |≤m1/2 1/2 − n n j=1 j |Xj |≤mn n j=1

1 X 2 h 1 X i2 ' X − Xj → VarX a.s. n j=1 j n j=1 n

n

Part (a) of Theorem 1.3 follows from Part (a) of Theorem 1.1 because σn2 → VarX a.s. by the law of large numbers. Next we prove part (b) of Theorems 1.1 and 1.3. This requires some preparation. The first lemma tells us that the limit in (1.4) must be normal. 1.8. Lemma. If the limit (1.4) holds a.s. then the L´evy measure of the limit µ(ω) is zero a.s. and n X

I|Xj |>λan = 0 eventually a.s. and n Pr{|X| > λan } → 0

(1.21)

j=1

for all λ > 0. Proof. By a.s. infinitesimality, µ(ω) is a.s. infinitely divisible. Let π(ω) be its L´evy measure. First we will show that π is non–random (a constant measure on a set of probability 1). By the converse clt (e.g. Araujo and Gin´e, 1980, Chapter 2), with probability one ¯ ¡ ∗ ¢¯ (ω) ¯|x|>δ →w π(ω)¯|x|>δ (1.22) nL∗ Xn,1 for all δ = δ(ω) such that π(ω){δ, −δ} = 0. Since we cannot control the continuity points of the possibly uncountable number of masures π(ω), we must smooth these measures out. For each δ ∈ Q+ , let hδ be a bounded, even, continuous function identically zero on [0, δ/2] and identically one on [δ, ∞), and set πδ (dx, ω) = hδ (x)π(dx, ω). Then, (1.22) implies n X

hδ (Xi /an )δXi /an →w πδ a.s.

j=1

Let F be a countable measure determining class of real bounded continuous functions on R (e.g. F = {cos tx, sin tx : t ∈ Q}). Then, the previous limit gives that, on a set of measure one, Z n X f dπδ a.s. for all f ∈ F and δ ∈ Q+ . hδ (Xi /an )f (Xi /an ) →w j=1

12 Since an → ∞ and the summands are bounded, the sum of the first k termsRat the left hand side is a.s. eventually zero for all k, and therefore, the variables f dπδ are all measurable for the tail σ-algebra of the sequence X. Since there are only a countable number of them, there is a common set of probability one where they are all constant, by the zero–one law. Since F is measure determining, πδ (ω) is a constant a L´evy measure π such that ¯ measure ¯ on this set for all δ. Hence, there is−1 ¯ ¯ πδ (ω) |x|>δ = π |x|>δ a.s. for all δ > 0. Let π ¯ = π ◦ |x| . (1.22) then becomes n X

I|Xj |>λan → π ¯ (λ, ∞)

(1.23)

j=1

for all λ > 0 of continuity for π ¯ , in particular on a countable dense set D of R+ . (1.23) implies that π ¯ (λ, ∞) takes on only non–negative integer values for all λ ∈ D. Suppose π ¯ (λ, ∞) = r 6= 0 for some λ > 0. Then, n X

I|Xj |>λan = r eventually a.s.,

j=1

which in particular implies n ©X ª lim Pr I|Xj |>λan = r = 1.

n→∞

(1.24)

j=1

On the other hand, there is enough uniform integrability in (1.23) to have convergence of expected values (e.g., by Hoffmann–Jørgensen’s inequality: see Lemma 1.12, Chapter 2) so that ª © n Pr |X| > λan → r and therefore,

n ©X rr lim Pr < 1, I|Xj |>λan = e−r n→∞ r! j=1

contradiction with (1.24). Hence, r = 0, that is, π = 0 and the limits (1.21) hold true. 1.9. Lemma. If the limit (1.4) holds a.s. then, the standard deviation σ(ω) of the normal component of the limit measure µ(ω) in (1.4) is a.s. a constant σ different from zero. If EX 2 < ∞, then n/a2n → σ 2 /VarX, whereas if EX 2 = ∞, then lim

n→∞

n X X2 i 2 a i=1 n

= σ 2 a.s.

(1.25)

Proof. The first limit in (1.21) gives that for all λ > 0 and p ∈ R, n X j=1

|Xj |p I|Xj |>λ = 0 eventually a.s.

(1.26)

13 and therefore, we can ‘untruncate’ in the necessary condition for the clt in terms of truncated variances, which then becomes · Pn 2 ³ Pn Xj (ω) ´2 ¸ j=1 Xj (ω) j=1 √ lim = σ 2 (ω) a.s. − (1.27) n→∞ a2n an n Since an → ∞, (1.27) shows that σ 2 (ω) is a tail random variable, thus a.s. constant, say σ 2 . Since µ(ω) is not degenerate with positive probability and π = 0 a.s., it follows that σ 6= 0. And of course, the fact that µ(ω) exists implies that σ < ∞. If EX 2 < ∞, the conclusion of the lemma follows from (1.27) and the law of large numbers. Let us now assume EX 2 = ∞. Then, we claim £Pn ¤2 j=1 |Xj |/n Pn lim = 0 a.s. (1.28) 2 n→∞ j=1 Xj /n (1.28) follows from the Paley–Zygmund argument applied to the empirical measure, that is, from the following self–evident inequalities, 1X 1X |Xj | ≤ a + |Xj |I|Xj |>a n j=1 n j=1 n

n

≤a+

n ³1 X

n

j=1

Xj2

n ´1/2 ³ 1 X

n

´1/2 I|Xj |>a

,

j=1

¡Pn ¢1/2 2 upon dividing by and then taking limits first as n → ∞ and then j=1 Xj /n as a → ∞ (use the Kolmogorov law of large numbers, both for finite and infinite expectations, when taking these limits). Inequality (1.28) was stated as a lemma (with other powers, besides 1 and 2) in Gin´e and Zinn (1989) and J. Chen informed us some time after publication that he and H. Rubin already had noticed this inequality, with a different proof, in Chen and Rubin (1984). Zinn and I hereby acknowledge their priority (we did not have the opportunity to publish an acknowledgement before). Back to the proof, it is clear that (1.25) follows from (1.27) and (1.28). Had we assumed the sequence an to be regular (precisely, an /n monotone, and an /n1/r monotone increasing for some r < 2), a refinement of Feller of the Marcinkiewicz law of large numbers (e.g., Stout, 1974, page 132) would automatically imply that (1.25) with 0 < σ 2 < ∞ and EX 2 = ∞ can not both be true at the same time, which would conclude the proof of Theorem 1.1, part b, by contradiction. But without any conditions, an extra argument is needed. Here it is: 1.10. Conclusion of the proof of Theorem 1.1 b). Assume EX 2 = ∞. By (1.26), we can truncate the variables Xj at an in (1.25), Lemma 1.9, and then take expectations of the resulting sums –by Hoffmann–Jørgensen’s inequality (see Chapter 2, Lemma 1.12 below), on account of the boundedness of the summands– to obtain n EX 2 I|X|≤an → σ 2 6= 0. (1.29) a2n

14 This inequality implies, by the monotonicity of the an ’s, that lim sup n→∞

n a2k < ∞. max a2n k≤n k

Since n/a2n → 0, (1.29) also implies that there exists rn → ∞ such that δn :=

n max a2 → 0. a2n k≤rn k

Also, since by the first limit in (1.21), Lemma 1.8, |Xn |/an < 1 a.s., the Borel– P∞ Cantelli lemma gives n=1 Pr{|X| > an } < ∞, and therefore, ∞ X

k Pr{ak−1 < |X| ≤ ak } → 0.

k=rn

Using (1.29) once more, together with the last three limits, we obtain n n X 2 ak Pr{ak−1 < |X| ≤ ak } n→∞ a2 n

σ 2 ≤ lim

k=1

≤ lim δn + lim sup n→∞

n→∞

∞ n£ a2k ¤ X k Pr{ak−1 < |X| ≤ ak } → 0. max a2n k≤n k k=rn

This contradicts σ 2 6= 0 and therefore we must have EX 2 < ∞ and an '

√ n.

1.11. Proof of Theorem 1.3 a). It has already been observed above that £ ∗ ¤ ¯ n )/σn √n 2 = 1/n, −X EX 2 < ∞ implies (1.7). Assume (1.7) holds. Since E∗ (Xn,1 the random variable at the left of (1.7) is conditionally a.s. the n-th row sum of an infinitesimal triangular array of independent variables. Then, a.s. asymptotic normality implies (by the converse clt) ¯ ¯ n ¯X ∗ − X o ¯n¯ n,1 √ nPr∗ > ε → 0 a.s., σn n for all ε > 0, that is,

n X

I|Xj −X¯ n |/σn √n>ε → 0 a.s.

j=1

for all ε > 0. This shows ¯ | |Xj − X √ n → 0 a.s. 1≤j≤n σn n max

(1.30)

If EX 2 < ∞ there is nothing to prove. If EX 2 = ∞ then we can combine (1.28) with (1.30) to conclude that |Xj | max pPn

1≤j≤n

i=1

Xi2

→ 0 a.s.

15 But this can only happen if EX 2 < ∞ by a result of Kesten (1971). 1.12. Remark on the proofs of Theorem 1.2 and Theorem 1.3 b). By a subsequence argument, Theorem 1.2 a) follows by showing that the necessary conditions for normal convergence (the analogues of (1.17)–(1.19)) hold in probability. Then, to show that these limits hold, one uses the fact that, by the converse clt, 2 n Pr{|X| > δan } → 0 and na−2 n EX I|X|≤an → 1. For instance the analogue of (1.17) that must be proved in this case is n X

I|X|>δan → 0 in pr.

i=1

and for this, one just notices that E

n X

I|X|>δan = n Pr{|X| > δan } → 0.

i=1

It is not difficult to complete the proof. To prove the converse (part b), working as in the proof of Theorem 1.1 along subsequences, one obtains that, assuming EX 2 = ∞, P n 2 2 2 i=1 Xi /an → σ 6= 0 in probability. Then, the converse weak law of large numbers 2 2 gives, in particular, n Pr{|X| > δan } → 0 and na−2 n EX I|X|≤δan → σ for all δ > 0 which, by the direct clt, implies that X is in the domain of atraction of the normal law, with norming constants an . Theorme 1.3 b), direct part, follows from 1.2 a) and Raikov’s theorem, as observed above. As for part b, the studentized bootstrap clt in pr. gives, following the proof of Theorem 1.3 a) along subsequences, that the limit (1.30) holds in probability, a condition that is known to be necessary and suficient for X to belong to the domain of attraction of the normal law (O’Brien, 1980). 1.13. Convergence of moments in Theorem 1.1. We will only show that if VarX = 1 and Z is N (0, 1), then n n ¯X ¡ ∗ ¢ √ ¯¯o © ª ¯ ¯ Xn,i − Xn / n¯ → E exp t|Z| a.s. E exp t¯ ∗

(1.30 )

i=1

for all t > 0 (since (1.5’) has a similar proof). We assume without loss of generality that EX = 0 and that Z is independent of X. Let {εi } be a Rademacher sequence ∗ independent of {Xn,i }. Then, convexity, the properties of Rademacher variables Pn and the facts that maxj≤n Xj2 /n → 0 a.s. and j=1 Xj2 /n → 1 a.s., give ¢ ½ ¯ Pn ¡ ∗ ½ ¯ Pn ∗ ¯¾ ¯ n ¯¯¾ − X X ¯ ¯ i=1 εi Xn,i ¯ n,i i=1 ∗ ∗ √ √ E exp t¯ ¯ ≤ E exp 2t¯ ¯ n n ½ Pn ∗2 ¾ i=1 Xn,i ∗ 2 ≤ 2E exp 4t n · X n 2 2 o¸n n 4t Xj 2 1 =2 exp → 2e4t . n j=1 n

16 That is, the sequence of bootstrap exponential moments is a.e. a bounded sequence. By the bootstrap clt and the continuous mapping theorem, there is a.s. weak convergence of the conditional laws of ¢ ½ ¯ Pn ¡ ∗ ¯ n ¯¯¾ ¯ i=1 Xn,i − X √ exp t¯ ¯ n © ª to the law of exp t|Z| and, therefore, (1.3’) follows by a.s. uniform integrability.

1.14. Remarks on Theorems 1.4 and 1.5, the stable case. A simple sufficiency proof of Theorem 1.4 consists in routinely checking the conditions of the clt for triangular arrays, as in the normal case. Just to show how the limit mn /n → 0 gets into the picture, we will check one of these three conditions, concretely © ∗ ª > δamn → cδ p in pr. (1.31) mn Pr∗ Xn,i for all δ > 0, assuming that X is in the doa of a stable law with norming constants an , thus, in particular, assuming the necessary condition n Pr{|X| > δan } → cδ p for all δ > 0. The expected value of the left side of (1.31) satisfies h

∗

E mn Pr

©

∗ Xn,i

> δamn

ªi

=E

n hm X n

n

i

ª © I|Xj |>δamn = mn Pr |X| > δamn → cδ p

j=1

whereas its variance tends to zero: n ³m X ¡ ª¢´2 © n I|Xj |>δamn − Pr |X| > δamn E n j=1

ª´2 © m2n ³ E I|Xj |>δamn − Pr |X| > δamn = n © ª¢ © ª m2 ¡ = n 1 − Pr |X| > δamn Pr |X| > δamn ³nm ´ ª n mn Pr{|X| > δamn → 0 ' n ª beacuse mn /n → 0 and mn Pr{|X| > δamn is bounded. This proves (1.31). The remaining conditions for stable convergence are proved similarly. For the converse, just note that, if mn0 /n0 → c > 0 for some subsequence {n0 }, then the argument in the second part of the proof of Lemma 1.8 shows that the L´evy measure of the limit must be zero, which is not the case for a stable limit. The proof of Theorem 1.5 is more involved. Here we describe only part of the direct proof. The statement to be proved corresponding to (1.31), is n ª¢ © mn X¡ I|Xj |>δamn − Pr |X| > δamn → 0 a.s. n j=1

17 If the sequence mn is regular, it turns out that we can block, symmetrize and apply a maximal inequality as in the usual proof of the lil, to conclude that it suffices to show ½ ∞ 2n X © ª¢¯¯ ª m2n ¯¯ X¡ Pr − Pr |X| > δa I > ε < ∞, ¯ ¯ m n |X |>δa n i 2 2n j=1 n=1 and Prohorov’s exponential inequality shows that this is the case if (mn log log n)/n → 0. We omit the details. The proof of Theorem 1.6 is also omitted, and we turn now our attention to the proof of Proposition 1.7. This proof will illustrate how the bootstrap conditional distributions keep some of the skewness of the original distributions (this is not the case, obviously, for the normal approximation). 1.15. Proof of Proposition 1.7. We assume without loss of generality that EX = 0 and EX 2 = 1 (besides the crucial hypothesis E|X|3 < ∞). By translation invariance of the family of functions in the definition of d4 , we have · ³ Pn · ³ ∗ ∗ ¯ ´ ³ Pn Xi ´¸ ¯ n ´ ³ X ´¸ Xn,1 − X i=1 (Xn,i − Xn ) ∗ ∗ i=1 √ √ √ d4 L ,L ≤ nd4 L ,L √ . σn n n σn n n (1.32) (i) Let now f be as in the definition of d4 , that is, kf k∞ ≤ 1 and kf √ k∞ ≤ 1, 1 ≤ i ≤ 4. ∗ ¯ n )/(σn n) are respectively −√ X The first and second conditional moments of (Xn,1 0 and 1/n, just as the first and secon moments of X/ n. McLaurin’s development of f then gives ¯ ¯ ¯ ³ ∗ ³ X ´¯ ¯n ´ 1 ¯ ∗ ∗ ¯ ∗ Xn,1 − X ¯ 3 3 3¯ ¯ √ √ − Ef ¯E (Xn,1 − Xn ) − σn EX ¯ ¯E f ¯≤ σn n n 6σn3 n3/2 h i 1 ∗ 000 000 ∗ 3 ¯ + 3 3/2 E |f (η1 ) − f (0)||Xn,1 − Xn | 6σn n i h 1 000 000 3 + 3 3/2 E |f (η2 ) − f (0)||X| 6σn n := In + IIn + IIIn , (1.33) √ ∗ ¯ ηi , i = 1, 2, being random n,1 − Xn )/(σn n), √ variables respectively between 0 and (X000 and between 0 and X/ n. Now, since σn → 1 a.s., supkf 000 k∞ ≤1 |f (ηi )−f 000 (0)| ≤ 2 and supkf (4) k∞ ≤1 |f 000 (ηi ) − f 000 (0)| ≤ |ηi | → 0 a.s., it follows that n3/2

IIIn → 0

(1.35)

IIn → 0 a.s.

(1.36)

sup kf 000 k∞ ≤1,kf (4) k∞ ≤1

and, by the law of large numbers, that n3/2

sup kf 000 k∞ ≤1,kf (4) k∞ ≤1

In is the crucial term in (1.33). In the analoguous proof of normal approximation, the term In would just be EX 3 /(6n3/2 ) but here it is of a smaller order because (σn3 − 1)EX 3 → 0 a.s.

18 and, by the law of large numbers, Pn n ¯ ¯ ¯1 X ¯ 2 ¡ 3 ¢ ¯ ∗ ∗ ¯ 3 3¯ 3 3¯ i=1 Xi ¯ ¯ ¯ + 2Xn ¯ → 0 a.s., X − EX − 3Xn ¯E (Xn,1 − Xn ) − EX ¯ = ¯ n j=1 i n which gives n3/2

sup kf 000 k∞ ≤1,kf (4) k∞ ≤1

In → 0 a.s.

(1.37)

combining the estimates (1.35)–(1.37) with (1.34) and then with (1.35), gives · ³ Pn ∗ ¯ ´ ³ P n X i ´¸ ¢ ¡ i=1 (Xn,i − Xn ) ∗ i=1 √ √ ,L = o n−1/2 , d4 L σn n n proving the proposition.

1.2. The general exchangeable bootstrap of the mean. For each n ∈ N, let wn = (wn (1), . . . , wn (n)) be a vector of n exchangeable random variables independent from the sequence {Xi } and satisfying the following conditions: Pn E1. wn (j) ≥ 0 for all n and j, and j=1 wn (j) = 1; ¢ ¡ E2. Var wn (1) = O n−2 . √ E3. max1≤j≤n n|wn (j) − 1/n| →P 0. ¢2 Pn ¡ E4. n j=1 wn (j) − 1/n →P c2 > 0. Define ¯ n∗ = X

n X

wn (j)Xj

j=1

¯ n . Newton and Mason (1992) proved and take this as the bootstrap of the mean X the following theorem. 2.1. Theorem. If EX 2 = σ 2 < ∞ and the weights wn are independent from the sample {Xi } and satisfy conditions E.1 to E.4, then ³√ ¡ ¢¯¯ ´ ¢ ¡ ∗ ¯ ¯ L n Xn − Xn ¯X →w N 0, c2 σ 2 a.s. Their original proof is based on a clt for exchangeable random variables due to H´ ajek. Here, following Arenal and Matr´ an (1996), we will deduce it simply from the usual Lindeberg clt for independent random variables, as follows: we will prove first convergence of the laws of the bootstrap variables conditioned on the weights, from which unconditional convergence will follow, and then we will show that unconditional convergence implies convergence of these laws conditioned on the sample. We may assume, without loss of generality that our random variables are defined on a product probability space, that the X’s depend on ω and the weights on ω 0 , so that the conditional laws given the weights or given the sample have a very specific meaning. Since BL(R) is separable, the dBL distance between one such conditional law and a fixed probability measure on R is measurable. The theorem

19 will follow from a series of simple lemmas. The assumptions in the lemmas are the same as those in the theorem (although some of the lemmas require less). 2.2. Lemma.

³√ ¡ ¢¯ ´ ¢ ¡ ¯∗ − X ¯ n ¯¯wn →w N 0, c2 σ 2 in probability n X n

L

in the sense that ³ ³√ ¡ ¢¯ ´ ¡ ¢´ ¯∗ − X ¯ n ¯¯wn , N 0, c2 σ 2 → 0 in probability. dBL L n X n Proof. By conditions E3 and E4, for every subsequence there is a further subsequence, call it n0 , such that ³ max 0

1≤j≤n

√

0

n0 |wn0 (j)

0

0

− 1/n |, n

n X ¡

wn0 (j) − 1/n0

¢2 ´

→ (0, c2 ) a.s.

j=1

Let ω 0 be a sample point for which this convergence takes place. Then the random variables Yn0 ,i =

(w 0 (j, ω 0 ) − 1/n0 )Xi 0 0 0 qP n ¡ ¢2 , i = 1, . . . , n , n ∈ {n }, n0 0 0 0 σ j=1 wn (j, ω ) − 1/n

form a triangular array of random variables which are i.i.d. by rows, and satisfy n X

0

VarX Yn,i = 1 and

i=1

lim 0

n →∞

n X

EX Yn20 ,i I|Yn0 ,i |>ε = 0 for all ε > 0.

i=1

Pn0 Then, by Lindeberg’s clt, i=1 Yn0 ,i converges in law to N (0, 1). Hence, ³ ³√ ¡ ´ ¡ ¢¯¯ ¢´ ∗ 2 2 0 ¯ ¯ → 0 a.s. dBL L n Xn0 − Xn0 ¯wn0 , N 0, c σ and the lemma follows. ¡√ ¡ ∗ ¢¢ ¢ ¡ ¯ −X ¯ n →w N 0, c2 σ 2 . n X n

2.3. Lemma. L

Proof. By Lemma 2.2, ³ ³√ ¡ ¢¯¯ ´ ¡ 2 2 ¢´ ∗ ¯ ¯ → 0 in probability. dBL L n Xn − Xn ¯wn , N 0, c σ Since these random variables are bounded by 2, convergence takes place in L1 as well, which gives ³ ³√ ¡ ¢´ ¡ ¢´ ¯∗ − X ¯ n , N 0, c2 σ 2 dBL L n X n h ³ ³√ ¡ ¢¯ ´ ¡ ¢´i ¯∗ − X ¯ n ¯¯wn , N 0, c2 σ 2 ≤ E dBL L n X → 0. n

20

Some preparation is necessary in order to condition with respect to the X’s in the above lemma. 2.4. Lemma. The sequence of conditional laws n ³√ ¡ ¢¯¯ ´o∞ ∗ ¯ ¯ L n Xn − Xn ¯X

n=1

is tight with probability one. Proof. The set Ω0 where this sequence is tight is Ω0 =

¯ 1o ³√ ¯ ¯ ¯ ∗ ¯ ¯ ¯ ¯ . ω : Prw n Xn (ω) − Xn (ω) ≥ N ¯ < k

∞ n ∞ \ ∞ [ \ k=1 N =1 n=1

¡ ¢ Now, taking into account that, by exchangeability and E1, Cov wn (i), wn (j) = −(Var wn (1))/(n − 1), and using E2, we have ³√ ¯ ´ ¯ ∗ ¯ ¯ ¯ ¯ Prw n Xn (ω) − Xn (ω) ≥ C n X ¡ ¢i n hX 2 ≤ 2 Xi (ω)Var wn (i) + 2 Xi (ω)Xj (ω)Cov wn (i), wn (j) C i=1 i 0 such that E|X1 |

2+δ

< ∞ and

∞ X k=1

δ

αk2+δ < ∞.

(3.9)

25 Condition (3.9) is stronger than (3.7). The main step in the proof of Theorem 3.1 consists in deriving a sort of Raikov Theorem (i.e. lln for squares) associated to the clt (3.1): with it we can control the relevant truncated bootstrap moments and thus derive the bootstrap clt from general principles (the criterion for convergence of row sums of infinitesimal arrays to a normal law already used above). The main tool is the basic covariance inequality of Davidov (1968). 3.1. Lemma. Let {ξn }∞ n=1 be a uniformly integrable sequence of real random variables. For each n ∈ N, let ξn,i , i = 1, . . . , n, be a strictly stationary set of random variables individually distributed as ξn . Suppose there exist constants an,i satisfying 1X an,i = 0 n→∞ n i=1 n

lim

and such that

(3.10)

¯ ¡ ¢¯¯ ¯ ¯Cov Y1 I|Y1 |≤M , Yi I|Yi |≤M ¯ ≤ M 2 an,i

(3.11)

for all M < ∞ and for all σ(ξn,i ) measurable random variables Yi . Then, 1 X¡ ξn,i − Eξn,i ) → 0 in pr. n i=1 n

(3.12)

Proof. Let Yn,i = ξn,i − Eξn,i . Since the sequence {Yn,1 }∞ n=1 is uniformly integrable, it follows that n ³ ¯1 X ´¯ ¯ ¯ ¯ ¯ E¯ Yn,i I|Yn,i |>Mn − EYn,i I|Yn,i |>Mn ¯ ≤ 2E¯Yn,1 ¯I|Yn,1 |>Mn → 0 n i=1

whenever Mn → ∞. For Mn → ∞ to be chosen below, set Y˜n,i = Yn,i I|Yn,i |≤Mn . It then suffices to prove that ¢ 1 X¡ ˜ Yn,i − EY˜n,i → 0 in probability. n i=1 n

Stationarity of the set Y˜n,1 . . . , Y˜n,n for each n, together with (3.11) (note Y˜n,i is σ(ξn,i ) measurable), gives n n¯ 1 X o ¡ ¢¯¯ 1 ¯ ˜ ˜ Pr ¯ Yn,i − EYn,i ¯ > ε ≤ 2 2 n i=1 n ε

X

¢ ¡ Cov Y˜n,i , Y˜n,j

1≤i,j≤n

¡ ¢¯¯ 2 X ¯¯ ≤ 2 ¯Cov Y˜n,1 , Y˜n,j ¯ nε

≤

j≤n n 2Mn2 X an,j nε2 j=1

26 for all ε > 0. Now, choosing Mn = to zero by (3.10).

¡Pn

¢−1/4

j=1 an,j /n

makes this probability tend

In Theorem 3.1 we can assume, without loss of generality, that EXi = 0, and we do so in what follows. For each n ∈ N and i = 1, . . . , N (n) := n − b(n) + 1, we let Zn,i be the sum of the data in block Bi , that is, Zn,i = Xi + . . . + Xi+b(n)−1 , n ∈ N, i = 1, . . . , N (n).

(3.13)

The previous lemma gives the following corollary, a kind of Raikov’s theorem associated to the clt in (3.1): 3.2. Corollary. Under the hypotheses of Theorem 3.1 and assuming EX = 0, we have: i) N (n) 1 X ³ Zn,i ´2 → 1 in pr., (3.14) N (n) i=1 σb(n) ii) for every δ > 0 and 0 ≤ p ≤ 2, N (n) ¯p k(n) X ¯¯ Zn,i ¯ √ → 0 in pr., ¯p ¯ I N (n) i=1 k(n)σb(n) |Zn,i |>δσb(n) k(n)

(3.15)

iii) N (n) 1 X Zn,i → 0 in pr., N (n) i=1 σb(n)

(3.16)

N (n) 1 X Zn,i √ I → 0 in pr., N (n) i=1 σb(n) |Zn,i |≤δσb(n) k(n)

(3.17)

and, for all δ > 0,

Proof. To prove that the limit (3.14) holds, we apply Lemma 3.1 with ξN (n),i = 2 2 Zn,i /σb(n) . First we note that, since b(n) → ∞, the clt (3.1) implies that the variables Zn,1 /σb(n) converge in law to a standard normal variable. Then, the second moments of these variables being all equal to 1, this gives that the sequence 2 2 2 {Zn,1 /σb(n) }∞ n=1 is uniformly integrable. Also, if X is σ(Zn,1 ) measurable and Y is 2 2 2 σ(Zn,i ) then, by the definition of αn , and since σ(Zn,1 ) ⊂ Fb(n) and σ(Zn,i ) ⊂ F i, we have that α(X, Y ) ≤ α(i−b(n))∨0 and that N (n) n−b(n)+1 X 1 b(n) 1 X + α(i−b(n))∨0 ≤ αi → 0 N (n) i=1 n − b(n) + 1 n − b(n) + 1

(3.18)

i=b(n)+1

(since αn → 0 and b(n)/n → 0). Davidov’s (1968) inequality (actually a particular case, Theorem 17.2.1 in Ibragimov and Linnik, 1971), to the effect that Cov(X, Y ) ≤ 4α(X, Y )kXk∞ kY k∞ ,

(3.19)

27 2 2 shows that the sequence {Zn,1 /σb(n) }∞ n=1 satisfies condition (3.11) with aN,i = 4α(i−b(n))∨0 . (3.18) gives condition (3.10) for these constants, and therefore, our sequence satisfies Lemma 3.1. Its conclusion, the limit (3.12), translates exactly into the limit (3.14). (Note that Lemma 3.1 also holds, with only the obvious changes, if the ξ variables are indexed by a sequence N (n) → ∞ of integers, instead of by N.) For every δ > 0 and 0 ≤ p ≤ 2, the array ¯ ¯p Zn,i ¯ ¯ √ , i = 1, . . . , N (n), n ∈ N, ζN (n),i = k(n)¯ p ¯ I k(n)σb(n) |Zn,i |>δσb(n) k(n)

also satisfies the hyptheses of Lemma 3.1 with respect to the same array of coefficients aN,i = α(i−b(n))∨0 as above: (3.11) certainly checks, and {ζN (n),1 }∞ n=1 is uniformly integrable because ζN (n),1 ≤ δ p−2 ξN (n),1 for all n. Now, the limit in (3.15) follows from Lemma 3.1 because the uniform integrability of the variables 2 2 /σb(n) implies Zn,1 ¯ ¯p Zn,1 ¯ ¯ √ lim k(n)E¯ p ¯ I n→∞ k(n)σb(n) |Zn,1 |>δσb(n) k(n) ¯ ¯2 Zn,1 ¯ ¯ √ ≤ δ p−2 lim E¯ p = 0. ¯ I n→∞ k(n)σb(n) |Zn,1 |>δσb(n) k(n) The rest of the statements follow in the same way, once we observe that the sequence {Zn,1 /σb(n) }∞ n=1 is uniformly integrable (since the sequence of its squares is) and EZn,1 = 0. Now the proof of Theorem 3.1 becomes a routine check of the classical conditions for normal convergence: ∗ Proof of Theorem 3.1. For each n, let Zn,i , i = 1, . . . , k(n), be an array of random variables which, conditionally on the sample {Xi }, are i.i.d. with (conditional) law © ∗ ª 1 , j = 1, . . . , N (n). = Zn,j = Pr∗ Zn,i N (n)

With this definition, we have ³ ¢ ˜ ∗ := L∗ p L H n ∗

¡

1 k(n)σb(n)

X¡

k(n)

∗ ∗ Zn,i − E∗ Zn,i

¢´

.

i=1

Hence, by previous arguments and the general criterion on convergence to the normal law of infinitesimal arrays, the proof of (3.4) reduces to checking that the following three limits hold for every δ > 0: ¯ ∗ n¯ o Zn,i ¯ ¯ Pr∗ ¯ p ¯ > δ → 0 in pr., k(n)σb(n) i=1

X

k(n)

28 X

k(n)

i=1

and

∗ ³ ´ Zn,i Var∗ p I|Z ∗ |≤δσb(n) √k → 1 in pr. k(n)σb(n) n,i

X

k(n)

i=1

∗ ³ ´ Zn,i E∗ p I|Z ∗ |>δσb(n) √k → 0 in pr.. k(n)σb(n) n,i

∗ By the definition of Zn,i , these three conditions can be written as: N (n) k(n) X √ → 0 in pr., I N (n) i=1 |Zn,i |>δσb(n) k

(3.20)

N (n) (n) ³ 1 NX ´2 1 X ³ Zn,i ´2 Zn,i √ √ I|Zn,i |≤δσb(n) k − I → 1 in pr. N (n) i=1 σb(n) N (n) i=1 σb(n) |Zn,i |≤δσb(n) k (3.21) and N (n) Z k(n) X p n,i I|Z |>δσb(n) √k → 1 in pr. (3.22) N (n) i=1 k(n)σb(n) n,i

Now, (3.20) and (3.22) are just (3.15) in Corollary 3.2 respectively for p = 0 and p = 1. The first and second terms at the left of (3.21) converge in probability respectively to 1 and to 0 by (3.14) and (3.17) in Corollary 3.2. Thus, we have proved that the limit (3.4) holds. Given (3.4), proving (3.3) reduces to showing that ¡ ∗ ¢2 σn → 1 in probability. 2 k(n)σb(n) ¡ ∗ ¢ ¡ ¢2 ∗ by conditional independence of the Zn,i variables, we Since σn∗ = k(n)Var∗ Zn,1 have ¡ ∗ ¢2 N (n) N (n) σn 1 X ³ Zn,i ´2 ³ 1 X Zn,i ´2 = − , 2 k(n)σb(n) N (n) i=1 σb(n) N (n) i=1 σb(n) which tends to 1 in probability by (3.14) and (3.16) in Corollary 3.2. Since the MBB procedure produces a triangular array of conditionally row– wise independent random variables, somehow with weaker dependence than original sample, it is conceivable that the MBB works in cases when the original clt does not. In fact this is the case, as shown by an example in Peligrad (1996, Remark 2.1). 1.4. The bootstrap of U–statistics. Degenerate U –statistics, together with the maximum of i.i.d. uniform variables, were among the first examples for which the regular Efron’s bootsrap (that is, sampling n times from the empirical measure Pn ) was seen not to work. Bretagnolle (1983) discovered that reduction of bootstrap

29 sample size makes the bootstrap consistent. These statistics constitute also an early example of the fact that one can often modify the bootstrap procedure so that it better simulates the original random mechanism. This may require, however, some information about the main features of the problem at hand. In the case of U – stastistics, a basic feature is the degree of degeneracy as it determines the OP size of the statistic. Arcones and Gin´e (1992) proposed to empirically degenerate the U –statistic to the same order as the original before bootstrapping: in this case no reduction of bootstrap sample size is necessary. We illustrate both ideas in a simple example. Let Xi be i.i.d., EX = 0, EX 2 = 1. Then, 1 n

X

Xi Xj →d Z 2 − 1,

2 (i,j)∈In

where Z is N (0, 1) and In2 = {(i, j) : 1 ≤ i, j ≤ n, i 6= j}. This statistic is degenerate of order 1. Let us write the statistic in the form · Pn ¸2 n 1X 2 1 X i=1 Xi √ Xi Xj = − X . n n i=1 i n 2 (i,j)∈In

A ‘naive’ application of the bootstrap gives · Pn n ∗ ¸2 1 X¡ ∗ ¢2 1 X i=1 Xn,i ∗ ∗ √ Xn,i Xn,j = − . X n n i=1 n,i n 2

(4.1)

(i,j)∈In

Now, the law of large numbers for X 2 bootstraps with no problem (the lln bootstraps): · Pn ¡ ∗ ¢2 · Pn ¡ ∗ ¢2 ¸ ¸2 ¸2 · Pn 2 ¡ ∗ ¢2 2 j=i Xj i=1 Xn,i i=1 Xn,i ∗ ∗ ∗ − 1 ≤ 2E − E Xn,1 −1 +2 E n n n ¸2 · Pn 2 4 ∗ ¡ ∗ ¢4 j=i Xj −1 ≤ E Xn,1 + 2 n n ¸2 · Pn n 2 4 X 4 j=i Xj − 1 → 0 a.s. = 2 X +2 n j=1 j n by the Marcinkiewicz and Kolmogorov laws of large numbers. But the clt part at the √ ¯right of identity (4.1) does not converge to a normal law because the centering nXn is missing. Bretagnolle’s solution was: reduce the sample size to make the missing centering go to zero (a.s. or in pr.) Obviously, taking the bootstrap sample √ ¯n, size in (4.1) to be mn (instead of n) turns the centering of the clt part into mn X which tends to zero in pr. if mn /n → 0 (by the clt), and tends to zero a.s. if (mn log log n)/n → 0 (by the lil). And this is what happens in general for the bootstrap of (the clt for) degenerate U –statistics: it works in pr if mn /n → 0 and it works a.s. if (mn log log n)/n → 0 (the rate mn (log n)1+δ /n → 0 was first used for the a.s. bootstrap but, as Arcones and Gin´e (1989) observed, (mn log log n)/n → 0 is the appropriate rate for the a.s. bootstrap in many situations –basically, those in which one can invoke some kind of lil). There is another logical solution to the above problem since, after all, one cannot ignore the centering in the bootstrap of the mean

30 even if EX = 0: just add the centering or, what √ is the same, reason this way: what makes the norming constants to be n instead of n in the Pclt for Xi Xj is that the X’s are centered, so we are in fact dealing with the statistic I 2 (Xi −EX)(Xj −EX)/n, n P ∗ ¯ n )(X ∗ − X)/n. ¯ −X And this works since which naturally bootstraps as In2 (Xn,i n,j 1 n

X ¡

¢¡ ¢ ∗ ¯n X ∗ − X ¯n −X Xn,i n,j

2 (i,j)∈In

· Pn ¡ =

i=1

¢ n ∗ ¯ n ¸2 1 X ¡ ∗ ¢ Xn,i −X ¯ n 2 →d Z 2 − 1 a.s. (4.2) √ − Xn,i − X n i=1 n

by the bootstrap clt and lln. This bootstrap (the ‘degenerate bootstrap’) has the disadvantage of requiring knowledge that we may not have (we may not know EX = 0), but it is useful in testing (we will elaborate on this below). We let (S, S) be a measurable space and P a probability measure on it, and let X, Xi be i.i.d. S–valued random variables with law P (i.e., the X’s do not have to be real). 4.1. Definition. A Pm –integrable function of m variables, f : S m → R, symmetric in its entries, is P–degenerate of order r − 1, 1 < r ≤ m, if Z Z m−r+1 f (x1 , . . . , xm )dP (xr , . . . , xm ) = f dPm for all x1 , . . . , xr−1 ∈ S whereas

Z f (x1 , . . . , xm )dPm−r (xr+1 , . . . , xm )

is not a constant function. If f is Pm –centered and is P–degenerate of order m − 1, that is, if Z f (x1 , . . . , xm )dP(x1 ) = 0 for all x2 , . . . , xm ∈ S, then f is said to be canonical or completely degenerate with respect to P. If f is not degenerate of any positive order we say it is non–degenerate or degenerate of order zero. In this definition the identities are taken in the almost everywhere sense. R With the notation P1 × · · · × Pm f = f d(P1 × · · · × Pm ), the Hoeffding projections of f : S m → R symmetric are defined as P f (x1 , . . . , xk ) := (δx1 − P) × · · · × (δxk − P) × Pm−k f πkP f (x1 , . . . , xk ) := πk,m

for xi ∈ S and 0 ≤ k ≤ m. Note that π0P f = Pm f and that, for k > 0, πkP f is a completely degenerate function of k variables. For f integrable these projections induce a decomposition of the U –statistic 1 Un (f ) := Un(m) (f ) := Un(m) (f, P) := ¡ n ¢ m

X 1≤i1 0 such that for all n large enough (depending on ω), ¯ √ ¯ ¯ √ √ ¯ √ c n|θn | ≤ n|H(θn )| ≤ n¯(Pn − P)h(·, θn )¯ + n¯Pn h(·, θn )¯ a.s.

109 Now, the last summand tends to zero a.s. by (6.12) (i.e., by definition) and the first is a.s. of the order of (log log n)1/2 by the law of the iterated logarithm for empirical processes over bounded P–Donkser classes –e.g., Dudley and Philipp (1983), Theorem 4.1. Therefore, ¶ µr log log n |θn | = O a.s. as n → ∞. (6.17) n Now, (6.17) and hypothesis (Z.3) (including (6.11)) allow us to apply Theorem 6.1 and conclude ¢¡ ¢¯¯ √ ¯¯¡ n¯ Pn − P h(·, 0) − h(·, θn ) ¯ → 0 a.s., so that, by (6.16), the non–bootstrap terms of the random variable at the left in (6.15) do indeed converge to zero a.s. Next, we deal with the bootstrap terms of (6.15): ´ √ ¡ ¢¡ ¢ √ ³ n Ph(·, θnb ) − Pbn h(·, 0) ' n Pbn − P h(·, θnb ) − h(·, 0) ¢ ¢¡ √ ¡ = n Pbn − Pn h(·, θnb ) − h(·, 0) ¢¡ ¢ √ ¡ n Pn − P h(·, θnb ) − h(·, 0) in prb , a.s. (6.16b) √ as n → ∞ since Ph(·, 0) = 0 ((Z.1)) and nPbn h(·, θnb ) → 0 in prb , a.s. Now, since by bootstrap consistency θnb → 0 in prb a.s. ((6.12) and (6.13)), applying the asymptotic equicontinuity condition associated to the bootstrap clt for for each ω fixed, we obtain ¢¡ ¢ √ ¡ lim n Pbn − Pn h(·, θnb ) − h(·, 0) = 0 in prb , a.s. n→∞

and it is the last summand in (6.16b) that requires Theorem 6.1, as before. For this, we need to estimate the size of θn . Using the full force of the differentiability condition (6.10), we find that for some c > 0 and for almost every ω, the following inequality holds with bootstrap probability tending to 1 as n → ∞: ¯ √ ¯ ¯ √ ¯ ¯ √ √ ¯ c n|θnb | ≤ n¯H(θnb )¯ ≤ n¯(Pbn − Pn )h(·, θnb )¯ + n¯(Pn − P)h(·, θnb )¯ ¯ √ ¯ + n¯Pbn h(·, θnb )¯. Now, the first summand is Oprb (1) a.s. by the bootstrap clt (a.s.); the last sumb b mand ³p is oprb (1)´ a.s. because θn → 0 in pr a.s.; and the second summand is (log log n) a.s. by the law of the iterated logarithm for empirical processes. O We conclude ¶ µr log log n |θn | = Oprb a.s. as n → ∞. (6.16b) n Then, as before, hypothesis (Z.3), (6.16b) and Theorem 6.1 give that ¯ √ ¯ n¯(Pn − P)h(·, θnb )¯ → 0 in prb , a.s. This shows that the bootstrap terms in (6.15) tend to zero in bootstrap probability a.s., concluding the proof of the theorem.

110 Theorem 6.3 applies to Huber’s (1964) location parameters with a significant simplification of the hypotheses (Arcones and Gin´e, loc. cit.). 6.4. Theorem. Let h : R → R be a bounded monotone function and let P be a probability measure on R. We let h(x, θ) := h(x − θ), θ ∈ R. Assume: (H.1) Letting H(θ) = Ph(·, θ), we have H(0) = 0, H 0 (0) = 1 and lim

r,s→0

(H(r) − H(s)) = H 0 (0). (r − s)

(6.18)

(H.2) There is a neighborhood U of 0 such that for all θ ∈ U , £ ¤2 P h(·, θ) − h(·, 0) ≤

c (log | log |θ||)1+δ

for some δ > 0 and c < ∞. (H.3) P is continuous on Cδ for some δ > 0, where Cδ denotes the open δ neighborhood of the set C of discontinuity points of h(x, θ(P )) = h(x, 0). Then

¡√

lim L

n→∞

¢ ¢ ¡√ n(θnb − θn ) = lim L nθn = N (0, VarP h) a.s. n→∞

(6.19)

where θn = inf{θ : Pn h(·, θ) > 0} and θnb = inf{θ : Pbn h(·, θ) > 0}. Proof. By (H.1), 0 is the only zero of the function H(θ). Since Pn h(·, ε) → H(ε) > 0 a.s. for all ε > 0 it follows that eventually θn ≤ ε a.s. Likewise θ ≥ −ε a.s., i.e. θn → 0 a.s.

(6.20)

The same argument using the bootstrap law of large numbers gives θnb − θn → 0 in prb a.s.

(6.21)

Let |θ| < δ/2. The sample points Xi satisfying Xi − θ ∈ Cδ/2 are all a.s. different by continuity of P on Cδ . Moreover, h(x) is continuous at x = Xi − θ if Xi − θ 6∈ Cδ/2 . Hence the function Pn h(·, θ) has a jump at θ of size at most 2khk∞ /n a.s. This proves, by (6.20) and (6.21), that √ √ nPn h(·, θn ) → 0 a.s. and nPbn h(·, θˆn ) → 0 in prb a.s. So, conditions (Z.4) and (Z.5) are satisfied. (H.1) is just (Z.1) and (Z.2). (H.2) is part of (Z.3). Finally the rest of (Z.3) is satisfied because of the monotonicity of h: any class of sets ordered by inclusion is ˇ Vapnik–Cervonenkis and therefore satisfies conditions (5.21) and (6.1) (see Example 5.2, 1)).

111 Theorem 6.3 and its corollary Theorem 6.4 contain the bootstrap CLT for the most usual M -estimators in particular for the median, Huber’s estimators, etc. For instance, Theorem 6.4 applies to h(x) = −kI(−∞,−k) (x) + xI[−k,k] (x) + kI(k,∞) (x) under minimal conditions on P, namely that P{k} = P{−k} = 0 and P(−k, k) 6= 0 (assuming Ph(·, 0) = 0).

References ´, E. (1978). On Poisson measures, Gausde Acosta, A., Araujo, A. and Gine sian measures and the central limit theorem in Banach spaces. In Probability on Banach spaces (J. Kuelbs Ed.), Advances in Probability and Related Topics, Vol. 4, pp. 1–68. Alexander, K. S. (1984). Probability inequalities for empirical processes and a law of the iterated logarithm. Ann. Probability 12 1041-1067. Alexander, K. S. (1985). Rates of growth for weighted empirical processes. In Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer (L.M. LeCam and R. A. Olshen, eds.) 2 475-493. Wadsworth, Monterey, CA. Andersen, N. T. (1985). The Calculus of Non–Measurable Functions and Sets. Various Publications Series, Aarhus Universitet, Matematisk Institut, No. 36. Aarhus. ´, V. (1987). The central limit theorem for stochastic Andersen, N. T. and Dobric processes. Ann. Probab. 15 164–177. ´, E. (1980). The central limit theorem for real and Banach Araujo, A. and Gine valued random variables. Wiley, New York. ´, E. (1989). The bootstrap of the mean with arbitrary Arcones, M. and Gine bootstrap sample size. Ann. Inst. H. Poincar´e 25 457-481. ´, E. (1991). Addition and Correction to ‘The bootstrap Arcones, M. and Gine of the mean with arbitrary bootstrap sample size’. Ann. Inst. H. Poincar´e 27 583–595. ´, E. (1992). On the bootstrap of U and V statistics. Ann. Arcones, M. and Gine Statist. 20 655–674. ´, E. (1992’). On the bootstrap of M-estimators and other Arcones, M. and Gine statistical functionals. In Exploring the limits of bootstrap (R. LePage and L. Billard eds.) pp. 13-48. Wiley, New York. ´, E. (1993). Limit theorems for U –processes. Ann. Probab. Arcones, M. and Gine 21 1494–1542. ˇ ´, E. (1994). U –processes indexed by Vapnik–Cervonenkis Arcones, M. and Gine classes of functions with applications to asymptotics and bootstrap of U –statistics with estimated parameters. Stochastic Proc. Appl. 52 17–38. ´rrez, E. and Matra ´ n, C. (1996). A zero–one law approach to Arenal–Gutie the central limit theorem for the weighted bootstrap mean. Ann. Probab. 24 532–540. Athreya, K. B. (1985). Bootstrap of the mean in the infinite variance case, II. Tech. Report 86–21, Dept. Statistics, Iowa State University. Ames, Iowa.

112 Bertail, P. (1994). Second order properties of a corrected bootstrap without replacement, under weak assumptions. Document de Travail # 9306, CORELA et HEDM. INRA, Ivry–sur-Seine. Bickel, P. J. and Freedman, D. (1981). Some asymptotic theory for the bootstrap. Ann. Statist. 9 1196-1216. ¨ tze, F. and van Zet, W. R. (1994). Resampling fewer than n Bickel, P., Go observations: gains, losses and remedies for losses. Preprint. Bretagnolle, J. (1983). Lois limites du bootstrap de certaines fonctionnelles. Ann. Inst. H. Poincar´e 19 281–296. Chen, J. and Rubin, H. (1984). A note on the behavior of sample statistics when the population mean is infinite. Ann. Probab. 12 256–261. ¨ rgo ˝ , S. and Mason, D. (1989). Bootstrapping empirical functions. Ann. Cso Statist. 17 1447–1471. Davydov, Y. A. (1968). Convergence of distributions generated by stationary stochastic processes. Theory Probab. Appl. 13 691–696. Deheuvels, P., Mason, D. and Shorack, G. R. (1992). Some results on the influence of extremes on the bootstrap. Ann. Inst. H. Poincar´e 29 83–103. Doukhan, P., Massart, P. and Rio, E. (1994). The functional central limit theorem for strongly mixing processes. Ann. Inst. H. Poincar´e 30 63–82. Dudley, R. M. (1978). Central limit theorem for empirical measures. Ann. Probability 6 899-929. Dudley, R. M. (1984). A course on empirical processes. Lecture Notes in Math. 1097 1-142. Springer, New York. Dudley, R. M. (1985). An extended Wichura theorem, definitions of Donsker class, and weighted empirical distributions. Lecture Notes in Math. 1153 141–178. Springer, Berlin. Dudley, R. M. (1987). Universal Donsker classes and metric entropy. Ann. Probab. 15 1306–1326. Dudley, R. M. (1989). Real Analysis and Probability. Wadsworth, Pacific Grove, California. Dudley, R. M. (1990). Nonlinear functions of empirical measures and the bootstrap. In Probability in Banach spaces, 7 pp. 63–82. (E. Eberlein, J. Kuelbs, M. B. Marcus eds.) Birkh¨ auser, Boston. Dudley, R. M. and Philipp, W. (1983). Invariance principles for sums of Banach space valued random elements and empirical processes. Z. Wahrsch. v. Geb. 62 509–552. Fernique, X. (1975). R´egularit´e des trajectoires des fonctions al´eatoires gaussiennes. Ecole d’Et´e de Probabilit´es de Saint Flour, 1974. Lecture Notes in Math. 480 1–96. 1995, Springer, Berlin. Filippova, A. A. (1961). Mises’ theorem on the asymptotic behavior of functionals of empirical distribution functions and its statistical applications. Theory Probab. Appl. 7 24–57. Gaenssler, P. (1987). Bootstrapping empirical measures indexed by Vapnikˇ Cervonenkis classes of sets. In Probability theory and Math. Statist. 1 467-481. VNU Science Press, Utrecht. ´, E. (1996). Empirical processes and applications: an overview. Bernoulli 2 Gine 1–28.

113 ´, E. and Zinn, J. (1984). Some limit theorems for empirical processes. Ann. Gine Probability 12 929-989. ´, E. and Zinn, J. (1986). Lectures on the central limit theorem for empirical Gine processes. Lecture Notes in Math. 1221 50-113. Springer, Berlin. ´, E. and Zinn, J. (1989). Necessary conditions for the bootstrap of the mean. Gine Ann. Statist. 17 684–691. ´, E. and Zinn, J. (1990). Bootstrapping general empirical measures. Ann. Gine Probability 18 851-869. ´, E. and Zinn, J. (1991). Gaussian characterization of uniform Donsker clases Gine of functions. Ann. Probability 19 758-782. ´, E. and Zinn, J. (1992). Marcinkiewicz type laws of large numbers and Gine convergence of moments for U –statistics. In Probability in Banach Spaces 8 273– 291. ¨ tze, F. (1993). Abstract in Bulletin of the IMS. Go Hall, P. (1988). Rate of convergence in bootstrap approximations. Ann. Probab. 16 1665–1684. Hall, P. (1990). Asymptotic properties of the bootstrap for heavy–tailed distributions. Ann. Probab. 18 1342–1360. Hoeffding, W. (1948). A non–parametric test of independence. Ann. Math. Statist. 19 546–557. Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. J. Amer. Statist. Assoc. 58 13–30. Hoffmann-Jørgensen, J. (1974). Sums of independent Banach space valued random variables. Studia Math. 52 159–189. Hoffmann-Jørgensen, J. (1984). Stochastics processes on Polish spaces, Aarhus Universitet, Matematisk Inst., Various Publication Series No. 39, 1991. Aarhus. Huber, P. J. (1964). Robust estimation of a location parameter. Ann. Math. Statist. 35 73-101. Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. Proceedings Fifth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1, 221-233. University of California Press, Berkeley. ´, M. and Janssen, P. (1993). Consistency for the generalized bootstrap Huskova for degenerate U –statistics. Ann. Statist. 21 1811–1823. Ibragimov, I. A. and Linnik, Y.V. (1971). Independent and Stationary Sequences of Random Variables. Wolters–Noordhoff, Groningen. Kahane, J. P. (1968). Some random series of functions. Heath, Lexington, Massachusetts. Kesten, H. (1971). Sums of random variables with infinite expectation. Solution to Advanced Problem # 5716. Amer. Math. Monthly 78 305–308. Klaassen, C. A. J. and Wellner, J. A. (1992). Kac empirical processes and the bootstrap. In Probability in Banach Spaces 8 411–429. (R. Dudley, M. Hahn, J. Kuelbs eds.) Birkh¨ auser, Boston. Kunsch, H. (1989). The jacknife and the bootstrap for general stationary observations. Ann. Statist. 17 1217–1241. ´ , S. and Woyczynski, W. (1992). Random series and stochastic inteKwapien grals: Single and multiple. Birkh¨ auser, Basel and Boston.

114 Le Cam, L. (1970). Remarques sur le th´eor`eme limite central dans les espaces localement convexes. A Les Probabilit´es sur les Structures Alg´ebriques, ClermontFerrand 1969. Colloque CNRS, Paris, 233-249. Ledoux, M. and Talagrand, M. (1986). Conditions d’integrabilit´e pour les multiplicateurs dans le TLC banachique. Ann. Probability 14 916-921. Ledoux, M. and Talagrand, M. (1988). Un crit`ere sur les petites boules dans le th´eor`eme limite central. Prob. Theory Rel. Fields 77 29-47. Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces. SpringerVerlag, New York. Liu, R. and Singh, K. (1992). Moving blocks jacknife and bootstrap capture weak dependence. In Exploring the limits of bootstrap pp. 225–248. (R. LePage and L. Billard eds.) Wiley, New York. Mason, D. and Newton, M. A. (1992). A rank statistics approach to the consistency of a general bootstrap. Ann. Statist. 20 1611–1624. Montgomery–Smith, S. J. (1994). Comparison of sums of independent identically dsitributed random variables. Prob. Math. Statist. 14 281–285. O’Brien, G. L. (1980). A limit theorem for sample maxima and heavy branches in Galton–Watson trees. J. Appl. Probab. 17 539–545. Peligrad, M. (1996). On the blockwise bootstrap for empirical processes for stationary sequences. Preprint. Pisier, G. and Zinn, J. (1978). On the limit theorems for random variables with values in the spaces Lp (2 ≤ p < ∞). Z. Wahrsch. v. Geb. 41 289–304. Politis, D. N. and Romano, J. P. (1994). Large sample confidence regions based on subsamples under minimal assumptions. Ann. Statist. 22 2031–2050. Pollard, D. (1984). Convergence of stochastic processes. Springer, Berlin. Pollard, D. (1985). New ways to prove central limit theorems. Econometric Theory 1 295-314. Pollard, D. (1990). Empirical processes: Theory and Applications. NSF–CBMS Regional Conference Series in Probability and Statistics, 2. IMS and ASA. Præstgaard, J. and Wellner, J. A. (1993). Exchangeably weighted bootstraps of the general empirical process. Ann. Probability 21 2053–2086. ´, D. (1996). The bootstrap of the mean for strong mixing sequences Radulovic under minimal conditions. Statist. and Probab. Lett. 28 65–72. ´, D. (1996b). On the bootstrap for empirical processes based on staRadulovic tionary observations. To appear in Stochastic Proc. Appl. ´, D. (1996c). Private communications. Radulovic Romano, J. P. (1989). Bootstrap and randomization tests of some nonparametric hypotheses. Ann. Statist. 17 141–159. Sen, P. K. (1974). On Lp –convergence of U –statistics. Ann. Inst. Statist. Math. 26 55–60. Sepanski, S. J. (1993). Almost sure bootstrap of the mean under random normalization. Ann. Probab. 21 917–925. Sheehy, A. and Wellner, J. A. (1988). Uniformity in P of some limit theorems for empirical measures and processes. Tech. Rep. # 134, Department of Statistics, University of Washington, Seattle, Washington.

115 Sheehy, A. and Wellner, J. A. (1988’). Almost sure consistency of the bootstrap for the median. Unpublished manuscript. Sheehy, A. and Wellner, J. A. (1992). Uniform Donsker classes of functions. Ann. Probability 20 1983-2030. Singh, K. (1981). On the asymptotic accuracy of Efron’s bootstrap. Ann. Statist. 9 1187–1195. Stout, W. F. (1974). Almost sure convergence. Cademic Press, New York. Strobl, F. (1994). Zur Theorie empirischer Prozesse. Doctoral Dissertation, University of Munich, Faculty of Mathematics. Stute, W. (1982). The oscillation behavior of empirical processes. Ann. Probab. 10 86–107. van der Vaart, A. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes. Springer, Berlin. Wellner, J. A. and Zhan, Y. (1996). Bootstrapping Z–estimators. Preprint. Yurinskii, Y. Y. (1974). Exponential bounds for large deviations. Theor. Probab. Appl. 19 154–155. Ziegler, K. (1994). On Functional Central Limit Theorems and Uniform Laws of Large Numbers for Sums of Independent Processes. Doctoral Dissertation, University of Munich, Faculty of Mathematics. University of Connecticut Department of Mathematics and Department of Statistics Storrs, CT 06269, USA