Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice [1 ed.] 0470510870, 9780470510872

Citation preview

Nonparametric Analysis of Univariate Heavy-Tailed Data

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

WILEY SERIES IN PROBABILITY AND STATISTICS

Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: David J. Balding, Noel A. C. Cressie, Nicholas I. Fisher, Iain M. Johnstone, J. B. Kadane, Geert Molenberghs, David W. Scott, Adrian F. M. Smith, Sanford Weisberg Editors Emeriti: Vic Barnett, J. Stuart Hunter, David G. Kendall, Jozef L. Teugels A complete list of the titles in this series appears at the end of this volume.

Nonparametric Analysis of Univariate Heavy-Tailed Data Research and Practice Natalia Markovich Institute of Control Sciences, Russian Academy of Sciences, Moscow, Russia

Copyright © 2007

John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777

Email (for orders and customer service enquiries): [email protected] Visit our Home Page on www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to [email protected], or faxed to (+44) 1243 770620. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 42 McDougall Street, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 6045 Freemont Blvd, Mississauga, ONT, L5R 4J3 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Anniversary Logo Design: Richard J. Pacifico

British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 978-0470-51087-2 Typeset in 10/12pt Times by Integra Software Services Pvt. Ltd, Pondicherry, India Printed and bound in Great Britain by TJ International, Padstow, Cornwall This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production.

To my parents and daughter

Contents

1

2

Preface

xi

Definitions and rough detection of tail heaviness 1.1 Definitions and basic properties of classes of heavy-tailed distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Tail index estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Estimators of a positive-valued tail index . . . . . . . . . . . . . . 1.2.2 The choice of k in Hill’s estimator . . . . . . . . . . . . . . . . . . . . . 1.2.3 Estimators of a real-valued tail index . . . . . . . . . . . . . . . . . . 1.2.4 On-line estimation of the tail index . . . . . . . . . . . . . . . . . . . . 1.3 Detection of tail heaviness and dependence . . . . . . . . . . . . . . . . . . . . 1.3.1 Rough tests of tail heaviness . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 Analysis of Web traffic and TCP flow data . . . . . . . . . . . . . 1.3.3 Dependence detection from univariate data . . . . . . . . . . . . . 1.3.4 Dependence detection from bivariate data . . . . . . . . . . . . . . 1.3.5 Bivariate analysis of TCP flow data . . . . . . . . . . . . . . . . . . . 1.4 Notes and comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 6 6 8 13 17 27 27 30 42 49 51 56 57

Classical methods of probability density estimation 2.1 Principles of density estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Methods of density estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Kernel estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Projection estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Spline estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.4 Smoothing methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.5 Illustrative examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Kernel estimation from dependent data . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Statement of the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Numerical calculation of the bandwidth . . . . . . . . . . . . . . . . 2.3.3 Data-driven selection of the bandwidth . . . . . . . . . . . . . . . . . 2.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Finance: evaluation of market risk . . . . . . . . . . . . . . . . . . . . .

61 61 70 70 74 76 76 83 85 86 89 91 91 91

viii

CONTENTS

2.4.2 Telecommunications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Population analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

93 94 95

Heavy-tailed density estimation 3.1 Problems of the estimation of heavy-tailed densities . . . . . . . . . . . 3.2 Combined parametric–nonparametric method . . . . . . . . . . . . . . . . . 3.2.1 Nonparametric estimation of the density by structural risk minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Illustrative examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Web data analysis by a combined parametric–nonparametric method . . . . . . . . . . . . . . . . . . . . 3.3 Barron’s estimator and  2 -optimality . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Kernel estimators with variable bandwidth . . . . . . . . . . . . . . . . . . . 3.5 Retransformed nonparametric estimators . . . . . . . . . . . . . . . . . . . . . 3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

99 100 101

109 111 113 117 119

4

Transformations and heavy-tailed density estimation 4.1 Problems of data transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Estimates based on a fixed transformation . . . . . . . . . . . . . . . . . . . . 4.3 Estimates based on an adaptive transformation . . . . . . . . . . . . . . . . 4.3.1 Estimation algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Analysis of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Further remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Estimating the accuracy of retransformed estimates . . . . . . . . . . . 4.5 Boundary kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Accuracy of a nonvariable bandwidth kernel estimator . . . . . . . . 4.7 The D method for a nonvariable bandwidth kernel estimator . . . 4.8 The D method for a variable bandwidth kernel estimator . . . . . . 4.8.1 Method and results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8.2 Application to Web traffic characteristics . . . . . . . . . . . . . . 4.9 The 2 method for the projection estimator . . . . . . . . . . . . . . . . . . . 4.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

123 123 124 128 128 129 133 135 136 139 141 142 142 144 147 149

5

Classification and retransformed density estimates 5.1 Classification and quality of density estimation . . . . . . . . . . . . . . . 5.2 Convergence of the estimated probability of misclassification . . 5.3 Simulation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Application of the classification technique to Web data analysis 5.4.1 Intelligent browser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Web data analysis by traffic classification . . . . . . . . . . . . . 5.4.3 Web prefetching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

151 151 154 155 160 160 161 161 161

2.5 3

103 107

CONTENTS

ix

6 Estimation of high quantiles 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Estimators of high quantiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Distribution of high quantile estimates . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Simulation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 Comparison of high quantile estimates in terms of relative bias and mean squared error . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Comparison of high quantile estimates in terms of confidence intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Application to Web traffic data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

163 163 164 167 169

7 Nonparametric estimation of the hazard rate function 7.1 Definition of the hazard rate function . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Statistical regularization method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Numerical solution of ill-posed problems . . . . . . . . . . . . . . . . . . . . . . 7.4 Estimation of the hazard rate function of heavy-tailed distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Hazard rate estimation for compactly supported distributions . . . . 7.5.1 Estimation of the hazard rate from the simplest equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.2 Estimation of the hazard rate from a special kernel equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Estimation of the ratio of hazard rates . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.1 Failure time detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.2 Hormesis detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Hazard rate estimation in teletraffic theory . . . . . . . . . . . . . . . . . . . . . 7.7.1 Teletraffic processes at the packet level . . . . . . . . . . . . . . . . . 7.7.2 Estimation of the intensity of a nonhomogeneous Poisson process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8 Semi-Markov modeling in teletraffic engineering . . . . . . . . . . . . . . . 7.8.1 The Gilbert–Elliott model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8.2 Estimation of a retrial process . . . . . . . . . . . . . . . . . . . . . . . . . . 7.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

179 180 182 185

208 210 210 212 217

8 Nonparametric estimation of the renewal function 8.1 Traffic modeling by recurrent marked point processes . . . . . . . . . . . 8.2 Introduction to renewal function estimation . . . . . . . . . . . . . . . . . . . . 8.3 Histogram-type estimator of the renewal function . . . . . . . . . . . . . . . 8.4 Convergence of the histogram-type estimator . . . . . . . . . . . . . . . . . . 8.5 Selection of k by a bootstrap method . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Selection of k by a plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 Simulation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 Application to the inter-arrival times of TCP connections . . . . . . .

219 220 221 224 225 228 232 234 245

169 170 175 176

187 188 188 193 197 199 200 207 207

x

CONTENTS

8.9 8.10

Conclusions and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

247 248

Appendices A Proofs of Chapter 2

251

B

Proofs of Chapter 4

253

C Proofs of Chapter 5

267

D Proofs of Chapter 6

271

E

Proofs of Chapter 7

275

F Proofs of Chapter 8

285

List of Main Symbols and Abbreviations

291

References

295

Index

307

Preface

Heavy-tailed distributions are typical of phenomena in complex multi-component systems such as biometry, economics, ecological systems, sociology, Web access statistics and Internet traffic, bibliometrics, finance and business. Typical examples of such distributions are Pareto, Weibull with shape parameter less than 1, Cauchy, and Zipf–Mandelbrot law. Heavy-tailed distributions have been accepted as realistic models for various phenomena: WWW session and TCP flow characteristics (e.g., sizes and durations), on/off-periods of packet traffic, file sizes, service time and input in queuing models, flood levels of rivers, major insurance claims, extreme levels of ozone concentration, high wind-speed values, wave heights during a storm, and low and high temperatures. Examples of applications can be found in the books by Embrechts et al. (1997), Adler et al. (1998), Coles (2001), Beirlant et al. (2004), Reiss and Thomas (2005), McNeil et al. (2005), and Castillo et al. (2006). In both populations of living individuals and inanimate objects such as automobile motors a common tendency has been discovered: the mortality risk for living objects (or the hazard rate for inanimate objects) decreases at infinity, which corresponds to heavy-tailed distributions (Yashin et al., 1996). Insurance company disasters caused by large claims, the overloading of computers by large files and of energy networks by strong deviations of weather and climate phenomena from the average behavior are rare and dangerous events. The methodology described in the book is therefore of current interest. The analysis of heavy-tailed distributions requires special methods of estimation because of their specific features: slower than exponential decay to zero, violation of Cramér’s condition, possible nonexistence of some moments, and sparse observations at the tail domain of the distribution. For example, the central limit theorem, which states the convergence of sums of independent and identically distributed (i.i.d.) random variables (r.v.s) to a Gaussian limit distribution, holds for a large variety of distributions: all we need is a finite variance of the summands.

xii

PREFACE

If this variance is infinite, then we get so-called stable distributions as limit distributions of the normalized sums (Lévy, 1925; Khintchine and Lévy, 1936). Cramér’s condition, which states the existence of the moment generating function, is violated for heavy-tailed distributions. Therefore, many results of the large deviation theory that require Cramér’s condition (e.g., Cramér’s theorem, which states the convergence of the tail of the finite sum of i.i.d. r.v.s to a Gaussian tail) are violated (Petrov, 1975). A linear approximation of the renewal function (RF) for large time intervals of observation changes for an infinite second moment as well. The statistical analysis of heavy-tailed distributions requires special methods that differ from classical tools due to the sparse observations in the tail domain of the distribution. For example, the histogram is a powerful tool of visual statistical data analysis. Small isolated bars often arise in histogram plots. The data which correspond to such bars are called ‘outliers’ and the compact mass of the bars is called the ‘body’ of the distribution. In classical textbooks the ‘outliers’ are considered as trash, deemed to be present in the sample as a result of some mistake. The usual recommendation is to remove them before any serious analysis or to use robust methods which are stable with respect to contamination of the data. But in many cases the ‘outliers’ are a vital part of the data; for example, the size of files transported by a network during the transfer of some firm’s home page may vary from kilobytes to megabytes (see Crovella et al., 1998). In a histogram large sizes will be viewed as apparent ‘outliers’. A network administrator who controls the operation of the network must take into account the existence of such files to avoid network overload. Theoretically, those data where the ‘outliers’ play a significant role are described by heavy-tailed distributions (Sigman, 1999). For compactly supported and light-tailed distributions (i.e., those without heavy tails) the histogram is a good estimate of the corresponding probability density function (PDF). But if the distribution is heavy-tailed, the histogram provides misleading peaks in the ‘tail’ domain or oversmoothes the ‘body’ of the PDF. The same is true for most of the common nonparametric PDF estimates such as ˇ kernel, projection and spline estimates (Cencov, 1982; Silverman, 1986; Devroye and Györfi, 1985). Usually, quantiles can be estimated by means of an empirical distribution function or weighted estimators based on sample order statistics. However, high quantiles (e.g., 99% or 99.9%) cannot be calculated in the usual way, since the empirical distribution function is equal to 1 outside the range of the sample. The hazard rate function decays to zero at infinity for heavy-tailed distributions, whereas it increases at infinity for light-tailed distributions and is constant for the exponential distribution. Hence, its estimation has to be different for various classes of distributions. Ignoring heavy tails in the data may lead to serious distortions of the estimation and errors in system control. This book focuses mainly on nonparametric methods of the statistical analysis of univariate heavy-tailed i.i.d. r.v.s from samples of moderate sizes. However, the

PREFACE

xiii

methods are widely useful for dependent data. Dependence detection, the estimation of the PDF from dependent data and elements of bivariate analysis are therefore also considered. The estimation of the PDF from empirical data is a central problem in mathematical statistics. The PDF is used for the description of the sample, classification, failure time detection, the construction of generators of random numbers, and the estimation of different functionals of the PDF such as the hazard rate function. The estimation of marginal distributions is the first step towards a multivariate analysis. Traditionally, two main sets of methods, the block maxima method and the peaks over thresholds (POT) method have been developed to estimate tail measures of the risk such as probabilities of exceeding high levels, high quantiles (called value-at-risk (VaR) in finance), and expected shortfall (Embrechts et al., 1997; Coles, 2001; Beirlant et al., 2004; McNeil et al., 2005). The block maxima (i.e., a set of maximal values selected in the blocks of data) are modelled by a generalized extreme value (GEV) distribution with distribution function (DF) Gx = exp− 1 + x − /−1/ . In the POT method the values + which are larger than some thresholds are modelled by the generalized Pareto . The parameters in distribution (GPD) with DF   x = 1 − 1 + x/−1/ + these models (in particular the tail index 1/) are estimated from a sample using nonparametric methods (e.g., Hill’s method) or parametric methods (e.g., maximum likelihood). In practice, we often need an estimate of the whole PDF or DF, both the ‘tail’ and the ‘body’, for example for classification or the estimation of the expectation. Another example is given by the copula technique (and, generally, multivariate analysis) which suggests the estimation of marginal distributions based on all data (Mikosch, 2006). The parametric tail models considered are not a good fit for the whole DF and the PDF and, hence, are not appropriate for such aims. Therefore, in this book, much attention is devoted to the nonparametric estimation of heavy-tailed PDFs. We consider three sets of estimators of the whole heavy-tailed PDF that are purely or partly nonparametric. These are variable bandwidth kernel estimators, combined estimators that fit the ‘tail’ and the ‘body’ of the PDF by parametric and nonparametric models respectively, and estimators based on the transformation approach. The need for different amounts of smoothing at different locations of heavytailed PDFs leads to the usage of kernel estimators with window width (or, roughly speaking, the ‘width’ of the kernel) varying from one point to another, that is, variable bandwidth kernel estimators (Abramson, 1982; Hall, 1992; Silverman, 1986). However, these estimators, at least with compactly supported kernels, are not intended for the estimation of a heavy-tailed PDF in the ‘tail’ domain, where the observations are sparse. This is because the latter estimators are defined on finite intervals. These are approximately the same as the ranges of the samples.

xiv

PREFACE

Application of heavy-tailed kernels for variable bandwidth kernel estimators has yet to be investigated in the literature. It is obvious that nonparametric PDF estimates with good behavior in the ‘tail’ domain are required. This feature is significant for classification (pattern recognition) purposes when the PDFs of many populations are compared. If one uses an empirical Bayesian classification algorithm, then the observations will be classified by the comparison of the corresponding PDF estimates of each class. Since the object can arise in the ‘tail’ domain as well as in the ‘body’, a tail estimator with good properties is of primary importance for classification. To improve the PDF estimation at infinity a transform–retransform scheme is considered here. This scheme implies a preliminary transformation of the data to a finite interval, that is, to a sample with a PDF that is more convenient for the estimation. Then one can estimate the PDF of a new r.v. obtained by the transformation by means of some nonparametric method and get the PDF of the original data by the reverse transformation of the PDF estimate of the transformed data. Furthermore, the back-transformed PDF estimates with fixed smoothing parameters work like location-adaptive estimates and allow the estimation of the PDF to be improved on the entire domain on which it is defined. Logarithmic transformations are a popular choice with this approach. In this book, combinations of data transformations and nonparametric estimates are considered that provide accurate PDF estimation and have decay rates at infinity close to those of the original PDFs. In this respect, a good deal of attention is devoted to a so-called adaptive transformation to a finite interval, which uses essentially the asymptotic distribution of the maximum of the sample as a model of the distribution behavior at infinity. The latter idea is followed throughout the book: an adaptive transformation may be applied to the PDF, and high quantile and hazard rate estimation to classification. A parametric–nonparametric estimation combines the advantages of parametric tail models to describe the ‘tail’ well enough and nonparametric methods to describe the ‘body’ domain (i.e., that limited area of relatively small values of an underlying r.v.) better. A similar idea was proposed in Barron et al. (1992), where a parametric model of the ‘tail’ of the PDF is superimposed on a histogram estimate of the ‘body’. Despite its ease of application, it is extremely sensitive to the correct choice of the parametric family and may provide a poor fit of the ‘body’ of a PDF in the case of moderate sample sizes. In practice, we often observe r.v.s governed by multimodal heavy-tailed distributions. Hence, it is important to use combined estimators aimed at accurately fitting both the multimodal ‘body’ and the ‘tail’ of the PDF. For practical needs, it is more important to provide such estimates of the PDF that are more suited to the tasks in hand. That is why another topic of the book concerns the investigation of the capacities of the PDF estimates considered with regard to the pattern recognition problem. Many methods of classification that use PDF estimates are known (Silverman, 1986; Aivazyan et al., 1989). We consider a procedure that allows increased influence of ‘outliers’ in the tail domain on the

PREFACE

xv

quality of the classification, thus preventing large misclassification losses by rare events. High quantile estimates for heavy-tailed distributions are applied to determine the values of characteristics of observed objects that may lead to rare but large losses. High quantiles indicate the VaRs in finance or the thresholds of parameters in complex systems such as the Internet (e.g., the 99.9% quantile can provide the maximal threshold for the file size) or atomic power stations. In this book, we discuss some but not all known high quantile estimators. The tail index is a key characteristic of heavy-tailed data. It shows the shape of the tail of the distribution without making any assumption regarding the parametric form of the tail. By means of the tail index, one can identify a heavy tail in measurements and the number of finite moments. All characteristics of heavy-tailed r.v.s are based on the tail index. In this book, many well-known estimators of the tail index such as Hill’s, POT, moment, UH, and ratio estimators are considered. Furthermore, a relatively new tail index estimator, proposed in Davydov et al. (2000) – called the group estimator here – is described. It has the essential advantage that it can be calculated recursively. The latter property is convenient for on-line estimation. The mortality risk function plays a significant role in population analysis. It is connected with the finding of causes of certain events in the population such as morbidity and mortality. This function is called the hazard rate if the reliability of technical systems is under investigation. Hitherto, most analysts have used the parametric approach for mortality risk estimation from empirical data. This means that before carrying out the estimation one decides what kind of function the mortality risk is expected to be. However, it might be difficult to describe the data by means of these models sufficiently accurately applying the cause factors as parameters. The parametric approach is problematic for the analysis of population processes by means of semi-Markov models when the intensity of the appearance of events is interpreted as an intensity of the transition from one state to another. An alternative approach is to use nonparametric models, when only general information about the estimated function is available. For the estimation of the hazard rate, however, the nonparametric approach is rarely used: in the literature, the preliminary estimation of the PDF and the DF by kernel or histogramtype estimators (Prakasa Rao, 1983) and regularized estimates (Stefanyuk, 1992) has been considered. One reason for this is a specific difficulty arising from the different asymptotic behavior of this function in the right-hand part of its domain for light- and heavy-tailed distributions. Hence, its estimation has to be different for various classes of distributions. In this book, the data transformation approach to a finite interval is considered to estimate the hazard rates corresponding to compactly supported distributions by nonparametric methods. The estimation of the hazard rate is presented as an inverse ill-posed problem involving Volterra’s integral equation, and a so-called regularization method, (Tikhonov and Arsenin, 1977) is used to find its approximation. The estimation of the hazard rate and the hazard rate ratio

xvi

PREFACE

is considered for a biological application (the problem of hormesis detection) and for teletraffic problems. For the purposes of warranty control, reliability analysis of technical systems, and particularly of telecommunication networks, one often needs to estimate the RF. This function is equal to the mean number of arrivals of the relevant events before a fixed time. Usually, measurement facilities count the events of interest, for example, the number of requested and transferred Web pages, incoming or outgoing calls in consecutive time intervals of fixed length. To estimate the RF, several realizations of the counting process (e.g., observations of number of calls over several days) may be required, with further averaging inside the corresponding time interval. However, it may be that the RF has to be estimated using only one set of inter-arrival times between events. This applies particularly to warranty control or when it would be too expensive to obtain numerous observations of the process. Explicit forms of the RF are obtained only for a few inter-arrival time distributions such as the uniform, exponential, Erlang or normal (Asmussen, 1996). The preliminary estimation of the DF or the PDF, if the latter exists, may become a more complicated problem than direct estimation of the RF. Here, the main attention is devoted to the nonparametric estimation of the RF from a sample of the i.i.d. inter-arrival times between events of moderate size. A few known results in this area (Frees, 1986a, 1986b; Grübel and Pitts, 1993; Schneider et al., 1990; Markovitch and Krieger, 2002b; Markovich and Krieger, 2006a) are discussed in this book. The well-known Frees estimate requires a huge amount of calculation even if one operates with samples as small as 20–30 observations. A sufficiently accurate estimate of the RF from empirical data is discussed that is also feasible for large samples. As always, the key problem of nonparametric estimates is the choice of the parameter that is responsible for the smoothing. Hence, the data-dependent selection of a smoothing parameter of the RF estimates is the main object of interest here.

The main methodology The statistical tools considered are based on the results of probability theory, mathematical statistics, extreme value theory, and the theory of the solution of ill-posed operator equations. The statistical methodology considered in this book is elaborated for the evaluation of characteristics of heavy-tailed r.v.s from samples of moderate size. Due to the lack of information beyond the range of the sample, nonparametric statistical estimation is based essentially on the asymptotic distribution of sample maxima as a model of the distribution behavior at infinity. The basic result of extreme value theory concerning the asymptotic behavior of the marginal distribution of the sample maxima (a GEV distribution) was provided by Gnedenko (1943). This result was extended to multivariate extreme value distributions by Galambos (1987).

PREFACE

xvii

The asymptotical tail distribution is the only realistic knowledge regarding the behavior of the distribution beyond the range of the sample. A data transformation approach that is discussed at length in the book essentially uses these asymptotic results. This approach allows us to transform the initial r.v. that is assumed to be GEV distributed into a new one. The latter may be located in a finite interval. That may both simplify the estimation (e.g., the estimation of the PDF) and allow us to apply some relevant estimators such as the histogram, or projection estimators that are applicable just for distributions with compact supports. The data transformations can be useful for the further development and the identification of models of multivariate distributions. It is known that such tools as copulas are invariant with respect to monotone transformations of r.v.s. That may give rise to construct dependence measures and models for ‘conveniently distributed’ r.v.s just using reliable transformations. Another methodology considered in the book is given by a statistical regularization method. This has evolved from Tikhonov’s regularization theory (Tikhonov and Arsenin, 1977). The latter theory was intended for the solution of deterministic linear and nonlinear operator equations. Due to the uncertainties in the availability of an operator and the right-hand part of the operator equation, the solution may be related to an ill-posed problem. Unlike Tikhonov’s method the method considered deals with stochastic operator equations. This approach was elaborated in Vapnik and Stefanyuk (1979), Vapnik (1982), and Stefanyuk (1986), and applied to population analysis in Markovich and Michalski (1995) and Markovich (1995, 2000) and to the analysis of teletraffic systems in Markovitch and Krieger (1999). Regularization is a developing area and is not restricted by the framework of Tikhonov’s scheme. The next step could be a wider application of other regularization schemes to statistical applications. In this book, the nonparametric estimation of characteristics of r.v.s plays a significant role. A smoothing of nonparametric estimates, for instance, the choice of the bin width in a histogram or the bandwidth in kernel estimators of the PDF, is key to an accurate approximation. The values of smoothing parameters recommended by theory usually minimize the mean squared error of the estimate or its asymptotic analog. This gives the values that are functions of a sample size. In practice, where one deals with samples of moderate sizes such values of parameters can provide unsatisfactory estimates. That is why, in this book, much attention is focused on data-dependent methods such as a cross-validation (Wahba, 1981) and the discrepancy method (Markovich, 1989; Vapnik et al., 1992). The stochastic version of the discrepancy method has evolved from the discrepancy method for deterministic operator equations (Morozov, 1984). Another approach is based on the minimization of an empirical bootstrap estimate of the mean squared error of the estimate by an unknown parameter. Bootstrapping is a tool for obtaining a reasonable value of an unknown smoothing parameter.

xviii

PREFACE

What is new? The book contains many results from the author’s advanced research material that are presented for the first time. These are: (i) the combined parametric–nonparametric estimator of a PDF; (ii) the adaptive data transformation that allows the PDF to be fitted at infinity better than a pure nonparametric estimate; (iii) the discrepancy method as a data-dependent smoothing tool of nonparametric PDF estimates; (iv) the application of the retransformed PDF estimates for classification; (v) on-line recursive estimation of the tail index; (vi) a modification of Weissman’s estimator of high quantiles that has smaller mean squared error; (vii) regularized estimates of the hazard rate function and hazard rate ratio; (viii) the estimator of the RF at finite time intervals from samples of inter-arrival times of moderate sizes; (ix) the bootstrap and plot methods as data-dependent smoothing tools for selecting a smoothing parameter in the RF estimator. Many practical recommendations for the implementation of the presented estimators are given, namely: (i) the use of nonparametric PDF estimates in finance, telecommunication, population analysis, and multivariate analysis; (ii) the usage of the classification methodology for the clustering of Internet data and Web prefetching; (iii) the usage of high quantile estimates in finance and the identification of parameter bounds in technical systems; (iv) the application of the hazard rate function in teletraffic (e.g., retrial call rate estimation); (v) the application of the hazard rate ratio in population analysis (e.g., hormesis detection) and for failure time detection; (vi) the application of RF estimates for overload control of telecommunication systems and warranty control; (vii) the rough detection of heavy tails and dependence in data and the application of these methods to Web traffic and TCP flow data by way of illustration.

PREFACE

xix

The reader can easily learn how to do a rough and more advanced statistical analysis of the data.

Content and general outline of the book The book gives a detailed survey of classical results and recent developments in the theory of nonparametric estimation of the PDF, the tail index, high quantiles, the hazard rate and the renewal function assuming the data come from i.i.d. random variables with heavy tails. Both asymptotic results such as convergence rates of the estimates and results for samples of moderate sizes supported by Monte Carlo investigation are considered. Special comments are also made on the application of the methods considered to dependent data. Observations that serve to clarify the main line of the exposition are located in footnotes. In Chapter 1 definitions and basic properties of classes of heavy-tailed distributions are considered. Tail index estimation and methods for the selection of the number of largest order statistics in Hill’s estimator are presented. Rough methods for the detection of heavy tails and the number of finite moments as well as dependence detection and simple bivariate analysis provide the ideas for a preliminary statistical data analysis. The methods considered are applied to measurements of Web traffic and TCP flows. Chapter 2 is devoted to PDF estimation. The main principles and the links between them are presented. Classical nonparametric estimators of the PDFs and smoothing methods are considered. PDF estimation using dependent data is discussed. Examples of the applications of PDF estimates are given. Chapter 3 describes three classes of heavy-tailed PDF estimation methods. These are methods that ‘paste’ together the parametric tail models and nonparametric estimates of the main part of the PDF (e.g., the combined parametric–nonparametric method and Barron’s estimator), the variable bandwidth kernel estimators, and the retransformed nonparametric estimators that use transformations of the data. In Chapter 4 so-called fixed and adaptive transformations are proposed. The difference between them is that fixed transformations do not depend on the distribution, in contrast to adaptive transformations. These transformations are applied to improve the estimation of heavy-tailed PDFs. Special boundary kernels are considered to improve the behavior of retransformed kernel estimates at infinity. The key problem of any nonparametric estimator is the choice of a smoothing parameter that determines the accuracy of the estimation. Data-dependent discrepancy methods are investigated both for nonvariable and variable bandwidth kernel estimators as well as for a projection estimator. The mean squared errors of these estimates are proved to be optimal. In Chapter 5 the application of the retransformed PDF estimates described in the previous chapter to the classification problem is considered. An empirical Bayesian algorithm is used. Then any new observation is classified by the comparison of the corresponding PDFs of each class. The retransformed kernel and polygram estimators are used to estimate heavy-tailed PDFs of each class. The accuracy of

xx

PREFACE

the classifiers obtained is compared by a simulation study. Possible applications of this classification technique to Web traffic data analysis and Web prefetching are considered. Chapter 6 contains estimators of the high quantiles for heavy-tailed distributions. The estimates are compared by a Monte Carlo study using simulated r.v.s. The distribution of the logarithm of the ratio of Weissman’s estimate to the true value of the quantile is proved to be asymptotically normal. The same result is obtained for the modification of Weissman’s estimate. An application to WWW traffic data is considered. Chapter 7 elaborates the nonparametric estimation of the hazard rate function in light- and heavy-tailed cases. The statistical regularization method and its theoretical background are presented. The application of the hazard rate and hazard rate ratio to telecommunication and population analysis is discussed. Finally, Chapter 8 includes the estimation of the renewal function within finite and infinite time intervals. Nonparametric estimators for finite intervals, their asymptotical theoretical properties and smoothing methods are considered. The companion website for the book is http://www.wiley.com/go/nonparametric

Audience This book is intended as a practical manual on the statistical theory of heavytailed data. The exposition is accompanied by numerous illustrations and examples motivated by applications in telecommunication, population analysis, and finance. Each chapter is provided with exercises. These may help the reader to understand the application of the statistical methods presented. The book assumes only an elementary knowledge of probability theory and statistical methods. Sometimes the subject requires the use of intermediate mathematical techniques such as probability theory, statistics, and mathematical analysis. The book is aimed at a relatively broad audience including students, practitioners, and engineers who are faced with analyzing heavy-tailed empirical data and are interested in the rough methodology and algorithms for numerical calculations related to the analysis of heavy-tailed data, as well as researchers and PhD students who are looking for new approaches and fundamental results, supported by proofs. Readers are expected to have diverse backgrounds including computer science, performance evaluation engineering, statistics, economics, demography, and population analysis. Readers with an interest in applied areas can skip the proofs of the theorems located in the appendices.

Acknowledgments This text was mainly developed in my habilitation thesis entitled ‘Estimation of characteristics of heavy-tailed random variables by limited samples’ and in a PhD course entitled ‘Analysis methods of heavy-tailed data’. It was given as a part of

PREFACE

xxi

the European project ‘EuroNGI. Design and engineering of the Next Generation Internet’. I am grateful to all my students for their corrections to the exercises. In particular, I would like to express my sincere thanks to Prof. A. Yagola of Lomonosov State University in Moscow, Prof. A. Kukush and Prof. R. Maiboroda of Shevchenko State University in Ukraine, and Dr. Sci. A. Dobrovidov and Dr. A. Stefanyuk of the Institute of Control Sciences, Moscow, for reviewing my habilitation thesis and useful discussions of my results. Prof. Dr. U. Krieger of Otto-Friedrich University, Bamberg, Germany, undertook to read the whole manuscript and made useful comments. I take pleasure in thanking Prof. P. Tran-Gia and Dr. Norbert Vicari of the Julius-Maximilians-University at Würzburg, Jorma Kilpi of VTT, Helsinki, who have provided me with teletraffic data. I express my special thanks to my PhD supervisor, Prof. V. Vapnik of Columbia University, who laid down the path for me to follow in my scientific life and the opportunity to become a working member of the Institute of Control Sciences of the Russian Academy of Sciences. The partial financial support I have received from the European research project ‘EuroNGI. Design and engineering of the Next Generation Internet’ and its continuation ‘EuroFGI’ (contracts no. 507613 and 028022) is also greately appreciated. I want to thank the entire Wiley team and, in particular, my copy-editor, Richard Leigh.

1

Definitions and rough detection of tail heaviness

In this chapter, the basic definitions and properties of heavy-tailed distributions are presented. Tail index estimation and methods for selecting the number of largest order statistics in the Hill estimator are discussed. Rough methods for the detection of heavy tails, the number of finite moments, dependence and long-range dependence are described. Elements of bivariate analysis are presented: estimation of the Pickands function and bivariate quantiles. The latter methods are applied to the analysis of telecommunication data.

1.1

Definitions and basic properties of classes of heavy-tailed distributions

We start with the common definitions. Definition 1 The set   P is called the probability space, where  is the space of elementary events,  is a -algebra of subsets of , and P is a probability measure on . Let   be some measurable space, R R be the real line with the -algebra R of Borelian sets on R.

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

2

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

Definition 2 The real-valued function X = X defined on  , is called a random variable (r.v.), if for any B ⊆ R  X ∈ B ⊆  holds. Definition 3 The function FX x = P X ≤ x , x ∈ R, is called the distribution function (DF) of the r.v. X. Definition 4 Let a nonnegative real-valued function ft, t ∈ R, exist such that for all x ∈ R,  x ftdt FX x = −

The function ft, t ∈ R, is called the probability density function (PDF) of r.v. X. Definition 5 The r.v.s X1  X2   Xn (Xi ∈ Bi ⊆ R, Bi is a finite set) are called independent if, for any x1  x2   xn ∈ R, PX1 = x1   Xn = xn = PX1 = x1 PXn = xn

or equivalently, for any B1   Bn ∈ R, PX1 ∈ B1   Xn ∈ Bn = PX1 ∈ B1 PXn ∈ Bn  In terms of DFs and PDFs, independence means that Fx1  x2   xn  = F1 x1 F2 x2  Fn xn  and fx1  x2   xn  = f1 x1 f2 x2  fn xn  where Fk xk  and fk xk  are the DF and PDF of the r.v. Xk . The definition of heavy-tailed distributions may be derived from the extreme value theory. Let X n = X1   Xn be a sample of independent and identically distributed (i.i.d.) r.v.s with DF Fx = PX1 ≤ x and Mn = maxX1  X2   Xn . It is known (Gnedenko, 1943; David, 1981) that if the limit distribution of maxima Mn exists then there exist normalizing constants an  bn such that PMn − bn /an ≤ x = F n bn + an x →n→ H x

x ∈ R

(1.1)

and an extreme value DF H x belongs to one of the following types of distribution function:1 ⎧ x > 0 > 0 (Fréchet) ⎨ exp−x−1/  x < 0 < 0 (Weibull) (1.2) H x = exp−−x−1/  ⎩ exp−e−x 

= 0 x ∈ R (Gumbel) The distribution H x can also be rewritten as  exp−1 + x−1/  = 0 H x = exp−e−x 

= 0

1

This result remains true if X1   Xn are weak dependent (Leadbetter et al., 1983).

(1.3)

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

3

where 1 + x > 0 (Jenkinson–von Mises representation). H x is called a standard generalized extreme value (GEV) distribution. Example 1 (Coles, 2001) If X n is a sequence of independent standard exponential r.v.s with DF Fx = 1 − exp−x for x > 0 then, letting an = 1 and bn = n in (1.1), the limit distribution of Mn as n →  is the Gumbel distribution. In the case of standard Fréchet r.v.s with DF Fx = exp−1/x and an = n and bn = 0, the limit distribution of Mn is precisely the standard Fréchet distribution with = 1 in (1.2). Let X n be a sequence of independent uniform r.v.s on 0 1 with DF Fx = x for x ∈ 0 1 and an = 1/n and bn = 1. Then the limit distribution of Mn is of Weibull type with = −1. Definition 6 The parameter is called the extreme value index (EVI) and defines the shape of the tail of the r.v. X. The parameter  = 1/ is called the tail index. Definition 7 We say that the r.v. X and its distribution F belong to the maximum domain of attraction of H x if (1.1) is fulfilled. We write X ∈ MDAH  (F ∈ MDAH ). We shall consider only nonnegative r.v.s. Definition 8 A DF Fx (or the r.v. X) is called heavy-tailed if its tail F¯ x = 1 − Fx > 0, x ≥ 0, satisfies, for all y ≥ 0, lim PX > x + y X > x = lim F¯ x + y/F¯ x = 1

x→

x→

This intuitively implies that if X exceeds a large value then it will most probably exceed any larger value, too. Roughly speaking, heavy-tailed distributions belong to the class of those long-tailed distributions whose tails decay to 0 slower than an exponential tail (Figure 1.1). The exponential distribution is often considered as a boundary between classes of heavy-tailed and light-tailed distributions. Typical examples of heavyand light-tailed distributions are given in Table 1.1. The class of heavy-tailed distributions comprises the subexponential class of distributions (S) and its subset, that is, distributions with regularly varying tails. Definition 9 The DF Fx (or the r.v. X), defined on 0 , is called subexponential (F ∈ S (X ∈ S)), if PSn > x ∼ nPX1 > x ∼ PMn > x

as x → 2

for some n ≥ 2, where Sn = X1 + + Xn , Mn = maxi=1 n Xi .

2

For any positive functions f and g, f ∼ g as x → x1 means that limx→x1 fx/gx = 1.

4

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

1–F(x)

1

0.5

0

0

2

4

6

8

10

x

Figure 1.1 Comparison of tail behavior: exponential distribution (solid line), Pareto distribution (dotted line).

Table 1.1 Examples of heavy- and light-tailed distributions. Heavy-tailed distributions

Subexponential: Pareto, lognormal, Weibull with shape parameter less than 1 With regularly varying tails: Pareto, Cauchy, Burr, Fréchet, Zipf–Mandelbrot law

Light-tailed distributions

exponential, gamma, Weibull with shape parameter greater than 1, normal, compactly supported distributions

Intuitively, subexponentiality means that the only way the sum can be large is by one of the summands getting large (in contrast to the light-tailed case, where all summands are large if the sum is so). Definition 10 The DF F (or r.v. X) is called a regularly varying distribution at infinity of index  = 1/ , > 0 (X ∈ R−1/ ), if PX > x = x−1/ ℓx ∀x > 0

(1.4)

where ℓx is called a slowly varying function (ℓx ∈ R0 ). Definition 11 A positive, Lebesgue measurable function ℓx on 0  is called a slowly varying function at infinity if limx→ ℓtx/ℓx = 1 ∀t > 0 (Feller, 1968; Sigman, 1999). Examples of ℓx are given by c ln x, c lnln x and all functions converging to positive constants. Using different functions ℓx, one can get a great variety of tails. For light-tailed distributions all moments EX + k  exist and are finite. In contrast, for regularly varying distributions the moments EX  are finite only if  < 1/ .

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

5

Basic properties of regularly varying distributions (Breiman, 1965; Bingham et al., 1987; Feller, 1971; Mikosch, 1999; Resnick, 2006) are summaryzed in the following lemma. Lemma 1

Let X ∈ R− . Then,

(i) X ∈ S. (ii) EX  <  if  < , EX  =  if  > .

(iii) If  > 1, then X r ∈ R1− and PX r > x ∼ ℓxx1− / − 1EX  as x → . (iv) If Y is nonnegative and independent of X such that PY > x = ℓ2 xx−2 , then X +Y ∈ R− min2  and PX +Y > x ∼ PX > x +PY > x as x → . (v) (Breiman’s theorem) If Y is nonnegative and independent of X such that EY + <  for some  > 0, then XY ∈ R− and PXY > x ∼ EY  PX > x

as x → 

Heavy-tailed distributions differ strongly from the normal or exponential distributions; for example, the exponential distribution function Fx = 1−e−x  x ≥ 0, satisfies F x + y/F x = exp−y

x ≥ 0

y ≥ 0

and hence it is not heavy-tailed. An important property of heavy-tailed distribution is given by the violation of Cramér’s condition. This means that the moment generating function does not satisfy Eex  < ,  > 0. Many results of the large deviation theory require the fulfillment of Cramér’s condition. Otherwise, for example, Cramér’s theorem on the convergence of PSn > x (Sn is the sum of n independent r.v.s) to the tail of a normal distribution is violated. Intervals of normal convergence of heavy-tailed distributions are presented in Mikosch and Nagaev (1998). ¯ In practice, a tail function Fx is often fitted by the generalized Pareto distribution. The latter is based on Pickands’ theorem (Pickands, 1975): Theorem 1 Let X1   Xn be an i.i.d. random sequence. The limit distribution of the excess of the Xi over the threshold u is necessarily of generalized Pareto form, lim

u↑xF u+x x X1 > u → 1 + x−1/ +

x ∈ R

where xF = supx ∈ R Fx < 1

is the right endpoint of the distribution Fx, the shape parameter ∈ R, and

6

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

x+ =

1.2



x 0

x > 0 x ≤ 0

Tail index estimation

The tail index reflects the shape of the distribution tail (with no assumption on the parametric form of the tail) and, therefore, plays a key role in the analysis of heavytailed measurements. The tail index is used for the estimation of high (99%, 99.9%) quantiles of observed r.v.s, the estimation of the PDF of the r.v. (Markovitch and Krieger, 2002a) and, hence, for classification (Maiboroda and Markovich, 2004). It allows one to identify roughly whether the distribution is heavy-tailed or not as well as to determine the number of finite moments. There are numerous estimators of the EVI . Let X n = X1   Xn be i.i.d. r.v.s with common DF Fx.

1.2.1

Estimators of a positive-valued tail index

Hill’s estimator for  = 1/> 0

We assume that Fx belongs to the class of regularly varying distributions (see Definition 10). For many applications, it is important to know . For example, if  < 2, than EX12 =  holds. Hill’s estimator (Hill, 1975), used for = 1/ > 0, is determined by 

H n k =

k 1 log Xn−i+1 − log Xn−k  k i=1

(1.5)

where X1 ≤ X2 ≤ ≤ Xn are the order statistics of the sample X n = X1  X2   Xn and k is a further smoothing parameter. It is a remarkable feature that the estimator (1.5) may be obtained in several ways – for example, by the maximum likelihood (ML) method assuming F ∈ R−1/ (Hill, 1975), by the regularly varying approach (de Haan, 1994), by the regression approach (Beirlant et al., 1999), or by using quantiles (Beirlant et al., 2004). For detailed discussion, see Embrechts et al. (1997) and Resnick (2006, Section 4.4). Hill’s estimator is weakly consistent if k → 

k/n → 0

as

n→

(Mason, 1982), and asymptotically normal with mean and variance 2 /k, √ H

n k − →d N0 2  k 

(1.6)

(Häusler and Teugels, 1985). In practice, the accuracy of the estimate depends on the selection of k. If the r.v. X ∈ R−1/ , then the slowly varying function ℓx, which is usually unknown, influences the estimation. Hill’s estimator does not work well if the r.v. X does not belong to class R−1/ . Plots of Hill’s estimates against k are

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS 18

9

16

8

7

1.5

1 7

14

6

12

0.5

5 10

0

8 6

gamma(k)

gamma(k)

gamma(k)

4 3

–0.5

2 –1

4 1 2

0

0

–1.5

–1 –2

–2 –4

–2

0

500 k

1000

–3

0

500 k

1000

–2.5

0

500 k

1000

Figure 1.2 Hill’s estimate against k for 15 realizations of the Weibull distribution (left), Pareto (middle) and Fréchet (right) distributions, each with parameter  = 05 (dotted line). The sample size is n = 1000.

shown in Figure 1.2 for 15 realizations of Weibull, Pareto and Fréchet distributions, each with parameter  = 05. The ratio estimator The ratio estimator an = an xn  =

n  i=1

lnXi /xn 1Xi > xn /

n 

1Xi > xn

(1.7)

i=1

is a generalization of Hill’s estimator in the sense that we use an arbitrary threshold level xn instead of an order statistic xn = Xn−k in (1.5) (Goldie and Smith, 1987). Here, 1A is the indicator function of the event A. The statistic (1.7) seems to

8

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

be among a few tail index estimators whose bias and mean squared error (MSE) asymptotics are known (Novak, 1996). Note that Hill’s estimator and the ratio estimator may also be applied to dependent data (Novak, 2002; Resnick and Stˇaricˇa, 1999). Hill’s estimator is very sensitive with respect to dependence in the data (see Ebmrechts et al., 1997). The asymptotic normality of the ratio estimator under the specific mixing condition that is fulfilled in many parametric models (e.g., ARCH and GARCH) is proved in Novak (2002).

1.2.2

The choice of k in Hill’s estimator

Visual choice of k The parameter k may be estimated visually by means of the exceedance plot, that is, the plot u eu X1 < u < Xn . Here eu =

n  i=1

Xi − u1Xi > u /

n 

1Xi > u

(1.8)

i=1

is the empirical mean excess function over threshold u of a given sample X n . The linearity of eu over some level u corresponds to a Pareto mean eP u = 1 + u/1 − . Then the number of the order statistic that is the closest to u is accepted as the estimate of n − k. Alternatively, one can estimate k from the Hill plot k ˆ H n k k = 1  n − 1 . The estimate of k is selected from the interval k−  k+  of stability of the function ˆ H n k. The latter approach is based on the consistency of Hill’s estimator. One may take the mean estimate (1.5) in k−  k+  as the estimate of

, that is, ˆ H n k ≈ for all k ∈ k−  k+ , and k corresponding to this as the optimal value. Methods of selecting k from empirical data are mostly based on the choice of a trade-off between the bias and the variance of Hill’s estimate. The bias increases and the variance decreases, as k increases. It was proved in Hall and Welsh (1985) that the asymptotical MSE of Hill’s estimate is minimal for  2 1/2+1 C  + 12 knopt ∼ n2/2+1  2D2 3 if the distribution function satisfies the so-called Hall’s condition

1 − Fx = Cx−1/ 1 + Dx−/ + ox−/  

Since parameters  > 0, C > 0 and D = 0 are unknown, this result cannot be applied directly to estimate k. Among adaptive procedures for the automatic choice of k one can mention the bootstrap methods (Hall, 1990; Danielsson et al., 1997; Caers and Van Dyck, 1999), which minimize the asymptotic MSE of the EVI, and the so-called sequential

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

9

procedure (Drees √ and Kaufmann, 1998), based on the fact that the maximal deviation of the statistic i ˆ H n i − , 2 ≤ i ≤ k, is of order log log n1/2 , that is, √ max i ˆ H n i − − bni  = Olog log n1/2  2≤i≤kn

in probability, for all intermediate sequences kn , where bni ∈ R are Hill estimator bias terms (Mason and Turova, 1994). Bootstrap method for selection of k The number k of retained data that are fitted to the tail corresponds to the minimum of the mean squared error (MSE), ˆ = E  ˆ − 2 = bias2   ˆ + variance  ˆ → min  MSE  k

Here the bias is given by bn k = E

H n k −  and the variance is determined by

2 H

H n k 

n k − E varn k = E 

ˆ We assume that Hill’s estimate 

H n k is used as . Since is unknown and MSE cannot be evaluated, the bootstrap approach proposes replacing in the MSE by an average calculated over some amount of resamples. These resamples are drawn from the initial sample X n randomly with replacement. This implies that some observations from X n will be represented in a resample with repetitions and others will not be represented at all. As a result, in order to estimate k one takes the value that minimizes a bootstrap empirical estimate of the MSE. More precisely, the bootstrap estimate of the bias is given by

∗H n1  k1  X n −

H n k b∗ n1  k1  = E and the bootstrap estimate of the variance is determined by



2

∗H n1  k1  − E

∗H n1  k1  X n X n  var ∗ n1  k1  = E  To construct these estimates, a smaller sample size n1 ≤ n is used and 

∗H n1  k1  =

k1 1  ∗ ∗ log Xn − log Xn 1 −i+1 1 −k1  k1 i=1 n

is Hill’s estimate of . It is determined by the resample X∗ 1 = X1∗   Xn∗1

∗ ∗ drawn randomly from X n with replacement, where X1 ≤ ≤ Xn are the order 1 n1 statistics of the sample X∗ . In the bootstrap estimates considered X n is fixed and

10

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

Figure 1.3 Classical bootstrap: resamples of the same size n as the sample X n are used (left). Nonclassical bootstrap: resamples of smaller size n1 = n  0 <  < 1, than n are used (right). n

the expectation is calculated among all theoretically possible resamples X∗ 1 . In practice, the expectation is replaced by the average over the underlying resamples. The reason for using smaller resamples is that the classical bootstrap with resamples of the same size n as the initial sample leads to underestimates of the bias. Using a smaller sample size n1 ≤ n and k1 data may help to avoid the situation where the bootstrap estimate of the bias is equal to zero regardless of the true bias of the estimate (Figure 1.3). Such situations arise particularly when linear estimates such as linear regressions or kernel estimates are used (Hall, 1990).3  Example 2 (Hall, 1990) Suppose ˆ is a linear function ˆ = ni=1 Xi  of  n data X1   Xn , and  ∗ = i=1 Xi∗  is the same function constructed from the  ˆ resample X1∗   Xn∗ . Then E ∗ X n = nEXi∗  X n = n ni=1 n−1 Xi  = , n ∗ since Xi may be selected in n ways from X . This implies that the bias of the ˆ −  = 0. bootstrap estimate is bias∗ = E ∗ X n − ˆ = 0, but the bias of ˆ is E

∗ Note that bias is random. Hence, it is not a bias in the usual sense. 3

It seems that the problems with the classical bootstrap are even greater. It is proved in Bickel and Sakov (2002) that the statistic an Fn  maxX1∗   Xn∗  − bn Fn  (where an  bn are normalyzed constants, see (1.1)) does not converge to H x for the bootstrap with resamples of size n. If resamples of smaller size n1 < n are used, n1 → , n1 /n → 0 and von Mises’ condition 1 fx → x 1 − Fx x→ is satisfied, then   an1 Fn  maxX1∗   Xn∗1  − bn1 Fn  → H x

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

It was proved in Hall (1990) that if the tail satisfies

1 − Fx = C0 x−1/ + C1 x−2/ + o x−2/

11

(1.9)

as x → , > 0, and k1 ∼ n2/3 then 1

H n1  k1  − 2  MSEn1  k1  = E

and its bootstrap estimate  1  k1  = E MSEn

∗H n1  k1  −

H n k2 X n

2

= E

∗H n1  k1  X n − 2

H n kE

∗H n1  k1  X n + 

H n k2 = b∗ n1  k1 2 + var ∗ n1  k1 

are close. The values of k1 and k are related by:   n  0 <  < 1 k = k1 n1

(1.10)

The value n1 is chosen as

(1.11)

n 1 = n 

The optimal value of k1 and, by (1.10), the optimal value of k are found by choosing  1  k1 . Since that value which minimizes the estimated mean squared error MSEn the DF Fx is unknown, one can estimate instead of the bias b∗ n1  k1  and variance var ∗ n1  k1  their empirical bootstrap estimates

and

B 1  bˆ ∗ n1  k1  =

∗H n1  k1  −

H n k B b=1 b

 2 B B 1  1 ∗H ∗H  n1  k1  =   var

n1  k1  −

n1  k1   B − 1 b=1 b B b=1 b ∗

respectively (Efron and Tibshirani, 1993). Here B denotes the total number of n1 sized bootstrapped resamples, and 

b∗H n1  k1  is Hill’s estimate derived from one of the resamples. Caers and Van Dyck (1999) recommended finding k1 by minimizing ∗

 n1  k1  MSE∗ n1  k1  = bˆ ∗ n1  k1 2 + var

(1.12)

Here all possible values of k1 , where k1 is an integer in the interval 1 n1 , are examined. Hall (1990) recommended taking  = 23   = 21 for Pareto-type distributions. Hence, the values of k satisfy (1.6). Caers and Van Dyck (1999) investigated these values of  and  using Monte Carlo simulations for a variety of distributions and found that they lead to the best results for the MSE. From our experience they provide good estimates of if the tails are sufficiently heavy, that is, the EVI should be large enough.

12

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

Double bootstrap method for the selection of k The double bootstrap, proposed in Danielsson et al. (1997), improves the bootstrap method (Hall, 1990), since it requires fewer parameters: one has to select n1 and B, but  in (1.10) is not required. Instead of estimation of the MSE we use the auxiliary statistic

2 MSEznk  = E znk − z⋆nk  where

znk = Mnk − 2 ˆ H n k2 

Mnk =

k

2 1 log Xn−j+1 − log Xn−k  k j=1

and z⋆nk is a bootstrap estimate of znk . Since Mnk /2 ˆ H n k and ˆ H n k are consistent estimates of , znk → 0 as n → . Therefore, the asymptotic MSE of znk is defined by AMSEznk  = Eznk 2 → mink . Hence, the value kˆ nopt of k, which minimizes AMSEznk , has the same order in n as knopt , the value that minimizes AMSE ˆ H n k. The double bootstrap algorithm is as follows: √ • Draw B bootstrap subsamples of size n1 ∈  n n (e.g., n1 ∼ n3/4 ) from the original sample and determine the value kˆ n⋆ 1 that minimizes the MSE of zn1 k . • Repeat this for B subsamples of size n2 = n21 /n (x is the integer part of x) and determine the value kˆ n⋆ 2 that minimizes the MSE of zn2 k . • Calculate kˆ nopt by the formula  2ˆ 2−1  ˆ n⋆ 2   k 1 1 1 kˆ nopt =  1− ˆkn⋆ ˆ 1 2

ˆ 1 =

log kˆ n⋆ 1  2 logkˆ n⋆ /n1  1

and estimate by Hill’s estimate with kˆ nopt . The method is robust with respect to the choice of n1 (Gomes and Oliveira, 2000). Sequential procedure for the selection of k Drees and Kaufmann (1998) provide the following algorithm for this procedure: √ • Obtain an initial estimate ˆ0 = ˆ H n 2 n for the parameter by Hill’s estimate. • For rn = 25 ˆ0 n025 , compute ˆ n  = mink ∈ 2  n − 1 max kr

2≤i≤k

√ H i ˆ n i − ˆ H n k > rn 

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

13

√ If rn is too large and max2≤i≤k i ˆ H n i − ˆ H n k > rn is not satisfied, it ˆ n  is well defined. is recommended to repeatedly replace rn by 09rn until kr ˆ n  for  = 07. • Similarly, compute kr • Calculate

ˆ opt

k

1 = 3



ˆ n  kr ˆ n  kr

1/1−

2 ˆ 0 1/3

and estimate by ˆ H n kˆ opt . The method is sensitive to the choice of rn .

1.2.3

Estimators of a real-valued tail index

Among nonparametric estimators, we mention the moment estimators (Dekkers et al., 1989) and UH estimators (Berlinet et al., 1998) that are not restricted to positive . They can be rewritten in terms of Hill’s estimator. The moment estimator is given by

ˆ M n k = ˆ H n k + 1 − 05 1 −  ˆ H n k2 /Snk  −1  (1.13)

2 k where Snk = 1/k i=1 log Xn−i+1 − log Xn−k . The UH estimator is determined by

ˆ UH n k = 1/k

k  i=1

log UHi − log UHk+1 

(1.14)

where UHi = Xn−i ˆ H n i. The kernel estimator of ,

+ n + i=1 i/nK i/n log Xn−i+1 − log Xn−i K 

ˆ n  =  1/n ni=1 K i/n

where  is a bandwidth parameter, log+ x = logx ∨ 1 and K is some nonnegative, nonincreasing kernel defined on 0  and integrating to 1, was proposed in Csörgo˝ et al. (1985). It is assumed that X0 = 1. This estimator generalyzes Hill’s estimator because the latter estimator may be obtained from the kernel estimator by the selection of the indicator function on the interval 0 1 as a kernel, that is, Ku = 10 < u < 1, and  = k/n. The Pickands estimator Xn−k+1 − Xn−2k+1 1 log  for k ≤ n/4 (1.15)

ˆ P n k = log 2 Xn−2k+1 − Xn−4k+1

has the same asymptotic properties (weak and strong consistency, asymptotic normality) as Hill’s estimator (Embrechts et al., 1997).

14

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

The estimator based on an exponential regression model essentially uses the approximate representation4 (in distribution, denoted by ≈D ) j log

Xn−j+1 − Xn−k Xn−j − Xn−k

≈D

E 1 − j/k + 1 j

j = 1  k − 1

where Ej , j = 1  n, denote standard exponential r.v.s (Beirlant et al., 2004), from which can be estimated by the ML method (Section 2.1). The resulting estimator is invariant with respect to a shift and a rescaling of the data. The POT method In the peaks-over-threshold (POT) method a generalized Pareto distribution (GPD)  1 − 1 + x/−1/  = 0  x = (1.16) 1 − exp −x/ 

= 0 where  > 0 and x ≥ 0, as ≥ 0; 0 ≤ x ≤ −/ , as < 0, is fitted to excesses over a high threshold. The method is based on the limit law for excess distributions (Balkema and de Haan, 1974; Pickands, 1975). Denote by Fu x = PX − u ≤ x X > u

(1.17)

the conditional distribution of the excess of X over the threshold u, given that u is exceeded. Pickands’ (1975) result states that condition (1.1) holds (F ∈ MDAH ) if and only if lim

sup

u→x+ 0 0 than the Hill and moment estimators. For < 0 the UH estimator is more efficient than the moment estimator: ⎧ 2 √ UH

d ⎨ N0  1 +   ≥ 0 1 − 1 + + 2 2  k ˆ n k − →  < 0 ⎩ N 0 1 − 2 (Caers and Van Dyck, 1999).

16

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

Under the conditions on Fx (F ∈ MDAH ) and Ux = F −1 1 − x−1 ,5 the Pickands estimator has the following property (see Dekkers and de Haan, 1989, p. 1799): 

 √ P

d

2 22 +1 + 1 k ˆ n k − → N 0  n →  2 2 − 1 log 22 For the estimator based on an exponential regression model, we have that   √ RMA  2

d n k − → N 0 2 k ˆ a

under some conditions on Ux and if we suppose that k n →  with k/n → 0 (see Matthys and Beirlant, 2003), where  1  1 1 − u + u log u 2 a = 2 du

0 1 − u and  2 equals the variance of K U with U uniformly distributed on (0,1) and K U  =

1+ 1 log u + 2 dilogu 

where dilogu =



1

u

log t dt 1−t

u ≥ 0

denotes the dilogarithm function. For the POT ML estimator,  MLP

Nu ˆ u − →d N0 1 + 2 

for Nu →  provided > −1/2, under the assumption that the excesses exactly follow a GPD (Smith, 1987). The asymptotic variances of the POT, PWM, EPM, and MOM estimators are given in Beirlant et al. (2004). Figure 1.4 compares the asymptotic variance for different estimators. The POT, Pickands, moment, and RMA n k estimators may suffer from substantial bias for some ill-behaving distributions like the loggamma, the lognormal and the inverse Burr distributions (Matthys and Beirlant, 2001). This bias is caused by violation of the necessary conditions on the tail required for the properties of normality considered. Sometimes it is difficult to compare a given estimator with others, since this estimator estimates some function of , but not itself; see, for instance, Davydov et al. (2000) or Section 1.2.4. In Csörg˝o et al. (1985) the asymptotic normality is

5

F −1 x denotes an inverse function.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

17

Asymptotic variance

20

15

10

5

0 –2

–1

0 gamma

1

2

√ Figure 1.4 Asymptotic variances of k ˆ −  for the Hill (solid line), moment (solid line with + marks), UH (solid line with circles), Pickands (dot-dashed line) and POT ML (dotted line) estimators. The variance of ˆ RMA is not represented, since it depends on the generator of uniform r.v.s. According to Matthys and Beirlant (2001), it nearly coincides with the asymptotic variance of the POT ML estimator for positive and it is lower than that of the moment estimator for negative . The UH and POT ML estimators coincide for

> 0.

√ proved for a more complicated statistic than k ˆ −  and does not allow one to represent the asymptotic variance and bias in such simple forms as before. It is mentioned in Polzehl and Spokoiny (2002) and Grama and Spokoiny (2003) that Hill’s estimator estimates not the tail index, but another parameter (see the trends in Figure 1.2) called the fitted Pareto index. This parameter is interpreted as the parameter of the Pareto distribution. In order to estimate this parameter, the authors propose a method based on successive testing of the hypothesis that the first k normed log-spacings follow exponential distributions with homogeneous parameters.6 Then the number k is chosen as the change-point detected.

1.2.4

On-line estimation of the tail index

In practice, on-line estimates are an important tool. At present several estimators of the tail index are known (see Sections 1.2.1 and 1.2.3). All these estimators are based on the order statistics and cannot be organyzed recursively. Here, the estimator that was proposed in Davydov et al. (2000) and investigated in Markovich (2005a) is considered. It is based on independent ratios of the second largest values to the 6

It is known that X = exp T +  is Pareto distributed, Fx = 1 − x − −a 

x ≥ 

with a = 1/, a ≥ 1, x > 1 + ,  ≥ 0 if T is exponentially distributed with parameter .

18

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

largest values of subsets of observations. The recursive behavior of this estimate is discussed. The bootstrap method for the estimation of its parameters is presented. Group estimator ( > 0) Let us consider a sample X n = X1   Xn of size n taken from a heavy-tailed DF Fx. We assume that X1   Xn are i.i.d. r.v.s. The tail index estimator considered in Davydov et al. (2000) has the essential advantage that it can be calculated recursively. According to this estimator the sample is divided into l groups V1   Vl , each group containing m random variables, that is, n = l · m. In practice, m is chosen and then l = n/m, where a denotes the integer part of a number a > 0. Let 1

Mli = maxXj Xj ∈ Vi

2

and let Mli denote the second largest element in the same group Vi . Let us denote 2

1

kli = Mli /Mli 

zl = 1/l

l 

kli 

(1.19)

i=1

Let a DF Fx satisfy the following relation as x → : 1 − Fx = x− ℓx

(1.20)

where  > 0, and ℓ is slowly varying, limx→ ℓtx/ℓx = 1. The distribution may satisfy the second-order asymptotic relation (as x → ) 1 − Fx = C1 x− + C2 x− + ox− 

(1.21)

with some parameters 0 <  <  ≤ , where C1 , C2 are positive constants.7 Then √ Davydov et al. (2000) proved that, for a distribution satisfying (1.20) and l = m =  n, zl →as

 1 =  +1 1+

and if Fx satisfies (1.21) with  = 2 then −1/2   l l l    −1 −1 2 −1 kli − 1 +  kli kli − l →p N0 1 l l i=1

i=1

(1.22)

(1.23)

i=1

  l √ −1  −1 kli − 1 +  l l →p N0  2 

(1.24)

i=1

with  2 =  + 1−2  + 2−1 as n → . One can construct confidence intervals using (1.23). Paulauskas (2003) also proved that (1.24) is valid for F satisfying 7

The case  =  corresponds to a Pareto distribution, and  = 2 to a stable distribution with 0 <  < 2.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

19

1/1+2 , where (1.21) not just for equal l and m, but if l = n n2/1+2 , m = −1 n n n → 0 as n →  and  =  − /. All these results are asymptotic, and any implementation for moderate sample size would in practice require additional research. In particular, since the parameters  and  are unknown, it is impossible to find l and m exactly. By (1.22) it follows that 1/zl − 1 can be used as an estimator of . We will call this estimator the group estimator. It is easy to see that (formula (7) in Paulauskas, 2003)     l l √ 1 √ 1  k − k − Ekl1 + bm (1.25) = l l l i=1 li 1 +  l i=1 li l √ 1  =√ kli − Ekl1  + lbm  l i=1

where bm is the bias of the estimator zl : Ezl = E

l  1 kli = Ekl1 = + bm  l i=1 1+

From Paulauskas (2003) we have that the MSE is given by     2  l l l 1 1  2 1 2 E kli − kli + var kli = l + bm2  = bias l i=1 1+ l i=1 l i=1 l

(1.26)

where l2 = Ekl1 − Ekl1 2 is the variance of kl1 . Then l and m are chosen in such a way that for the first term on the right-hand side of (1.25), the central limit theorem holds and the bias must stay bounded. In Paulauskas (2003) the upper bound for bm is obtained and the optimal l is given up to positive constants as the minimum of the upper bound of the MSE (1.26), l = On2/1+2 . The latter result is proved only for the distribution class (1.21). Subsequently, we do not assume that the shape of the distribution is known. The parameters l and m are selected by the bootstrap method, which is a nonparametric tool.  It is difficult to compare the estimator 1/l li=1 kli of /1 +  with other estimators, since this estimator estimates a different function of the tail index . One can only look at the asymptotic MSE of the estimates for known distributions with known parameters. We consider, for example, the ratio estimator (1.7). For Pareto distribution (1.21) we have MSEan  = E an / − 12  1 c2 2/2 −1 · 2 − 1c1 n−2 −1/2 −1 ∼ 2 − 1 1 −  c1 (1.27) (Novak, 2002) and

20

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

MSEzl  = Ezl − 1 + −1 2 ∼  2 /l ∼  2 n−2/1+2 =  2 n−2 −1/2 −1  (Markovich, 2005a). For the standard Cauchy distribution  = = 1,  = 3, we have MSEan  ∼ 5/416/811/5 /n4/5 (Novak, 2002) and MSEzl  ∼ 1/12n−4/5  (Markovich, 2005a). On-line estimation An on-line estimator may be defined to be one where each update, following the arrival of a new data value, requires only O1 (i.e., a fixed number) of calculations. In this respect, the recursiveness of the estimator is not necessary, although it is convenient. It is important for us to use the recursive property of the new tail index estimate. Suppose we get the next group of updates Vl+1 (this group should contain at least two but not more than m points). Denote the estimate of the EVI obtained by groups V1   Vl as l . Specifically, l obeys the equation l 1 1 k = l i=1 li 1 + l

since l = 1/ l 

(1.28)

Furthermore, we have (Markovich, 2005a)  −1  l+1 kl+1l+1 −1 1  1 l

l+1 = kl+1i · + − 1 −1 = l + 1 i=1 l + 1 1 + l l+1

and after getting i additional groups Vl+1   Vl+i with m elements each, −1  l − 1 + kl+1l+1 + + kl+il+i

l+i = l + i 1 + l

that is, l+i is obtained using l after O1 calculations. One can rewrite it as   i  zl+i = lzl + kl+jl+j /l + i j=1

Obviously, the bias of zl+i is the same as for zl , but the variance is less. In fact, 2 2 2 2 2 2 since 2 Ezl = Ek l1 , Ekli = Ekl1 , Ezi − Ezl = 1/i − 1/ll , and varzl  = 1/ll , l = varkl1  , we have varzl+i  = varzl l/l + i

Evidently, varzl+i  < varzl  for all i > 0.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

21

The number of operations to select two first maxima in the group with m elements is 2 log m (Knuth 1973, Section 5.2.3). We assume here that m is fixed. If real-time calculation is necessary, then it may not be practicable to recompute m whenever we believe that a very different value might be required, since the amount of recalculation would be too great. It would be more appropriate to update m with the new portion of data, relying on a long series of small changes to produce the large changes in m. Indeed, the accuracy of the tail index estimate is expected to be worse than it would be if m changed with each new portion of observations or if the number of observations in each group were not constant. As recommended in Paulauskas (2003), in practice one can plot m zm  m0 < n/m m < M0 , m0 > 2, M0 < n/2, where zm = m/n i=1 kn/mi (similarly to a Hill plot k ˆ H n k 1 ≤ k ≤ n − 1 ) and then choose the estimate of zm from an interval in which the function zm demonstrates stability. The background of this approach is provided by the consistency result (1.22) as n m → , m < n. Hence, there must be an interval m−  m+  such that zm ≈ /1 +  = 1 + −1 for all m ∈ m−  m+ . We suggest choosing the average value z = mean1/zm − 1 m ∈ m−  m+ 

(1.29)

and m∗ ∈ m−  m+  as a point such that zm∗ = z. In Figure 1.5 the plot m 1/zm − 1 is depicted for Pareto distributed samples with = 1; the true is shown by a dotted line. For n = 1000 we suggest taking z over the interval m−  m+  = 10 40. Then m∗ = 10 corresponds to the average z = 1087 and the estimate

ˆ = 0999.

8

1/z_m–1

6

4

2

0

0

20

40

60

80

100

m

Figure 1.5 Plot of m 1/zm − 1 for the Pareto distribution with = 1 and sample sizes n = 150 (dot-dashed line), n = 500 (dotted line), n = 1000 (solid line). The true is shown by horizontal solid line.

22

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

Bootstrap method to select m The automatic choice of m = n/l from empirical data could be done by minimizing the empirical bootstrap estimate of the mean squared error of 1 + l −1 , that is,  2 l 1 1 MSE l  = E k − → min  m l i=1 li 1 +

The bootstrap estimate is obtained by drawing B samples with replacement from the original data set X n . Some observations from X n may appear more than once, while others do not appear at all. One can use smaller resamples X1∗   Xn∗1 of the size n1 < n from X n to avoid the situation where the bootstrap estimate of the bias (or its asymptotic form) is equal to 0 regardless of the true nonzero bias of the estimator (Hall, 1990). The values n1 and n may be related by n 1 = nd 

0 < d < 1

The resample is divided into l1 subgroups and l1 = n1 /m1  holds. The size of subgroups m1 and m are related by: m = m1 n/n1 c 

0 < c < 1

(1.30)

Since the DF Fx is unknown, one can find m1 by minimizing the empirical bootstrap estimate of the MSE  2 ∗  l1  m1  MSE∗ l1  m1  = bˆ ∗ l1  m1  + var (1.31)

and use this value of m1 to calculate an optimal m using (1.30). All possible values of m1 , where m1 is an integer in the interval 2 n1 , are examined. Here, B 1 zb − zl  bˆ ∗ l1  m1  = B b=1 l1  2 B B  1 1  ∗  l1  m1  = zb − zb var B − 1 b=1 l1 B b=1 l1

are the empirical bootstrap estimates of the bias and the variance, respectively. l1 kl1i , constructed from some resample. Hence, an optimal We use zbl1 = 1/l1  i=1 l = n/m may be calculated and further used in (1.28) to estimate . The choice of suitable values of c and d is a problem. Based on asymptotic theory, Hall (1990) concludes that d = 1/2 and c = 2/3 lead to the most accurate results when the bootstrap estimation of the parameter by Hill’s estimate is considered. In what follows, we will tackle the problem by means of simulation. Taking into account the complicated proof technique related to the bootstrap approach we skip here the theoretical investigation of c and d for the group estimator. The number of iterations to find the minimum of (1.31) is defined by the required accuracy and an optimization method (Knuth, 1973).

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

23

Application to simulated data We investigate the influence of c for a fixed d by means of a Monte Carlo study. For this purpose, samples of the Pareto distribution with DF (4.8), Fréchet distribution with DF Fx = exp− x−1/ 1x > 0 

(1.32)

and Weibull distribution with PDF  sxs−1 exp−xs  x > 0 fx = 0 x ≤ 0

(1.33)

were generated. The latter distribution does not belong to class (1.20) or (1.21). The use of the statistics zl as estimators of the tail index for distributions other than (1.20) or (1.21) has yet to be theoretically investigated. Here, the application of the new estimator to the Weibull distribution is investigated by a simulation study. The values c ∈ 005 0101 05 are used for a fixed d = 05. In Figures 1.6–1.8 one can see that     NR NR  1 1 1 1   ˆ − 2  Bias =

ˆ i −  RMSE = 

NR i=1

NR i=1 i which are the relative bias and the square root of the mean squared error of the EVI estimator (1.28) calculated over NR = 500 repeated samples. The mean and standard deviation of the parameter m are also presented. B = 50 bootstrap resamples were taken. Samples of sizes n ∈ 150 500 1000 are considered. From the simulation study one can see that the best values of c for fixed d = 05 are 0.3, 0.4, 0.5. It is important to mention that the mean and the standard deviation of m behave rather similarly for c < 03 independent of the sample size. However, for c > 03 these values increase rapidly for larger n.

1

0

0.5

20 15

20

StDev m

Bias

RMSE

0.5

30

Mean m

1

10

10 5

–0.5

0

0.2

c

0.4

0.6

0

0

0.2

c

0.4

0.6

0

0

0.2

c

0.4

0.6

0

0

0.2

c

0.4

0.6

Figure 1.6 Simulation results of estimation for a Pareto PDF with = 1 and different c: 500 samples of n observations; n = 150 (solid line), n = 500 (dotted line) and n = 1000 (dashed line). Relative bias and mean squared error of the EVI estimator (first two plots on the left). Mean and standard deviation of the parameter m (last two plots on the right).

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS 0.4

RMSE

Bias 0

Mean m

0.3

0.2

0.2

15

15

10

10

StDev m

24

5

0.1 –0.2

0

0.2

c

0.4

0

0.6

0

0.2

c

0.4

0

0.6

0

0.2

c

0.4

5

0

0.6

0

0.2

c

0.4

0.6

Figure 1.7 Simulation results of estimation for Fréchet PDF with = 03 and different c: 500 samples of n observations; n = 150 (solid line), n = 500 (dotted line) and n = 1000 (dashed line). Relative bias and mean squared error of the EVI estimator (first two plots on the left). Mean and standard deviation of the parameter m (last two plots on the right).

4

40

4

60

2

StDev m

2

Mean m

RMSE

Bias

3 20

1 0

0

0.2

c

0.4

0.6

0

0

0.2

c

0.4

0.6

0

0

0.2

c

0.4

0.6

40

20

0

0

0.2

c

0.4

0.6

Figure 1.8 Simulation results of estimation for Weibull PDF with s = 05 and different c: 500 samples of n observations; n = 150 (solid line), n = 500 (dotted line) and n = 1000 (dashed line). Relative bias and mean squared error of the EVI estimator (first two plots on the left). Mean and standard deviation of the parameter m (last two plots on the right).

The confidence intervals for the estimate l with bootstrap estimated m are given in Table 1.2 for different heavy-tailed distributions, the levels p ∈ 0025 005 01 , and the best values of c ∈ 03 04 05 . It is assumed that the bootstrap estimates are normally distributed with mean and variance constructed from the set of bootstrap estimates 1∗   N∗ R , where NR is the number of bootstrap resamples. Then one can calculate the tolerant bounds of confidence intervals by the wellknown formula (Smirnov and Dunin-Barkovsky, 1965) u1  u2  = mean −  · StDev  mean +  · StDev  where mean and StDev are the mean and the standard deviation of NR bootstrap estimates. The interval is constructed in such a way that 1001 − p% of the distribution falls in this interval with probability P. The value  depends on NR , P, p and may be approximately represented by   5tp2 + 10 tp +  (1.34)  =  1 +  12NR 2NR

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

25

Table 1.2 Confidence intervals of the bootstrap estimates for heavy-tailed distributions and different c: 500 samples of n = 1000 observations each. PDF Pareto

=1

Fréchet

= 03

Weibull

= 05

c

mean (StDev )

p · 100%

Confidence interval

03

1191 0606

04

1141 0575

05

0964 0508

25 5 10 25 5 10 25 5 10

−0278 266 −0098 248 0115 2267 −0253 2535 −0082 2364 012 2162 −0268 2196 −0117 2045 0062 1866

03

03072 0155

04

03501 0175

05

03417 0172

25 5 10 25 5 10 25 5 10

−0069 0683 −0023 0637 0032 0583 −0074 0774 −0022 0722 0039 0661 −0075 0759 −0024 0708 0036 0647

03

0738 0371

04

05665 0291

05

0562 0296

25 5 10 25 5 10 25 5 10

−0161 1637 −0051 1527 0079 1397 −0139 1272 −0053 1186 005 1083 −0156 128 −0068 1192 0036 1088

Reprinted from Proceedings of 1st Conference on Next Generation Internet Design and Engineering, On-line estimation of the tail index for heavy-tailed distributions with application to WWW-traffic, Markovich NM, Table 1, © 2005 IEEE. With permission from IEEE.

Here,  is defined by the equation 1   −t2 /2 e dt = 20   = 1 − p (1.35) √ 2 − z 2 where 0 z = √12 0 e−t /2 dt is Laplace’s function. The DF of the standard normal distribution Nz 0 1 can be expressed as Nz 0 1 = 05 + 0 z for positive z. Furthermore, 0 −z = −0 z 0 0 = 0 0 − = −1/2, 0  = 1/2. The value tp is calculated by the equation

26

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

1   −t2 /2 e dt = 05 − 0 tp  = 1 − P √ 2 tp

(1.36)

For P = 099 we have 0 xp  = P − 05 = 049 and tp = 233. Furthermore, we have  ∈ 2245 197 1645 for p ∈ 0025 005 01 , respectively. In Table 1.2 mean and StDev are shown. Samples of size n = 1000 and NR = 500 were used. As before, B was taken equal to 50. From Table 1.2 it follows that the ratio r1 = mean / is closer to 1 for the Pareto and Fréchet distributions than for the Weibull distribution. This may imply the larger bias of the new estimator for the latter distribution. The ratios r2 = u2 / − 1 and r3 = u1 / − 1 corresponding to the upper and lower bounds u1 and u2 of the confidence interval are considered. The larger r2 and r3 are in absolute value, the wider is the confidence interval. From Table 1.2, r2 ∈ M M ∈ 0866 166 158 1581 1166 2274 , r3 ∈ L L ∈ −1278 −0885 −1258 −088 −1322 −0842 hold for Pareto, Fréchet p = 2.5%

2.5

2

1.5

1.35

1.5

0.3

0.4

1 0.2

0.5

0.3

0.4

c

c

p = 2.5%

p = 5%

0.5 0.2

0.5

r3

r3 1.25

0.95

0.4 c

0.5

1.1

1.06 0.2

0.4

0.5

p = 10%

0.9

0.85

1.08

0.3

0.3 c

1.12

1.3

1.2 0.2

1

r3

1 0.2

p = 10%

1.5

r2

2

r2

2

p = 5%

r2

2.5

0.3

0.4 c

0.5

0.8 0.2

0.3

0.4

0.5

c

Figure 1.9 Absolute values of ratios r2 and r3 for Pareto (solid line), Fréchet (dotted line), and Weibull distributions (dashed line), for confidence levels p ∈ 0025 005 01

and parameter c ∈ 03 04 05 . Reprinted from Proceedings of 1st Conference on Next Generation Internet Design and Engineering, On-line estimation of the tail index for heavytailed distributions with application to WWW-traffic, Markovich NM, Figure 5, © 2005 IEEE. With permission from IEEE.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

27

and Weibull, respectively. This implies that the confidence interval is worse for the Weibull distribution, at least with regard to the upper bound u2 . Figure 1.9 shows the comparison of r2 and r3 for different c and p. From this figure one may conclude that in most cases the values c = 04 and c = 05 correspond to the best values of u1 and u2 , respectively. The value c = 04 gives a cautious decision in the sense that r2 is the same irrespective of distribution and r3 is not maximized. In all cases, the Weibull distribution has the largest r2 and r3 . Together with the previous conclusion, this implies that the confidence intervals for this distribution are worse than for Pareto and Fréchet distributions.

1.3

Detection of tail heaviness and dependence

Before a serious analysis of the data is carried out, it is necessary to detect heavy tails in the data. For this purpose, nonparametric test procedures (e.g., Jureˇcková and Picek, 2001) or a set of rough statistical methods for heavy-tailed features (Embrechts et al., 1997; Markovich and Krieger, 2006b) can be applied. Here, we consider several simple procedures that may help us to detect heavy tails and the dependence structure of the data. We illustrate by means of real data how these methods are applied to analyze traffic measurements.

1.3.1

Rough tests of tail heaviness

Here, we consider several methods in order to check whether measurements X n = X1  X2   Xn are derived from a heavy-tailed DF Fx = PX1 ≤ x or not. We may also give rough estimates of the number of finite moments of the DF Fx. Ratio of the maximum to the sum Let X1  X2   Xn be i.i.d. r.v.s. We define the statistic (Embrechts et al., 1997, p. 308) Rn p =

Mn p  Sn p

n ≥ 1 p > 0

(1.37)

where Mn p = max  X1 p   Xn p  

Sn p = X1 p + Xn p 

n ≥ 1 (1.38)

to check the moment conditions of the data. Then the following equivalent assertions as

Rn p−→0

⇔ E X p < 

Rn p−→0

⇔

E X p 1 X ≤ x ∈ R0 

Rn p−→1

⇔

P X > x ∈ R0 

p

p

28

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

can be exploited. The class R− of distributions with regularly varying tails and the tail index  = 1/ , > 0, are defined in Definition 10. For different values of p the plot of n → Rn p gives some preliminary information about the distribution P X > x . Then E X p <  follows if Rn p is small for large n. For large n, a significant difference between Rn p and zero indicates that the moment E X p is infinite. Quantile–quantile plot The idea of the quantile–quantile (QQ) plot is to show the relationship    n−k+1 Xk  F −1 k = 1  n  n+1

where X1 ≥ ≥ Xn are the order statistics of the sample. A QQ plot is based on the following fact. It is well known that for an i.i.d. sample with a continuous DF Fx the r.v.s Ui = FXi 

i = 1  n

(1.39)

are independent and uniformly distributed on [0,1]. Then Xi = F −1 Ui . For example, if the exponential DF Fx is believed to be the distribution of X, then we plot exponential quantiles F −1 n − k + 1/n + 1 against the order statistics Xk of the underlying sample. Then F −1 x is an inverse function of the exponential DF and a linear QQ plot corresponds to the exponential distribution. Generally, one can investigate the quantiles of any distribution, not just an exponential. The linearity of a QQ plot shows that the parametric model of the distribution is selected correctly. Plot of the mean excess function To study the tail behavior in more detail, this simple test allows us to detect visually whether a tail is light or heavy. Let X be an r.v. with the finite right endpoint XF = supx ∈ R Fx < 1 of its support. Then eu = E X − u X > u 

0 ≤ u < XF ≤ 

(1.40)

is the mean excess function of the r.v. X over the threshold u. The sample mean excess function is defined by en u =

n  i=1

Xi − u1Xi > u /

n 

1Xi > u 

(1.41)

i=1

For heavy-tailed distributions the function eu tends to infinity. A linear plot u → eu corresponds to a Pareto distribution, the constant 1/ corresponds to an exponential distribution, and eu tends to zero for light-tailed distributions.8

8

The mean excess function is calculated by formula eu = 1/F u (1.40), (Embrechts et al., 1997, p. 162).

 XF u

F xdx, which follows from

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

29

60 15 50 10 40 e(u)

e_n(u)

5 0

30 20

–5 10 –10

0

5 u

10

0

0

5

10

15 u

20

25

30

Figure 1.10 Left: Mean excess functions for some distributions: exponential (horizontal solid line), lognormal with parameters (0,1) (dotted line), Pareto with shape parameter equal to 1 (upper solid line), Weibull with shape parameter equal to 0.5 (solid line) and 2 (dotdashed line). Right: Ten empirical mean excess functions en u, each based on simulated data of size n = 1000 from the Pareto distribution Fx = 1 − 1 + x/2−2 , x ≥ 0. A very unstable behavior, especially towards the higher values of u, can be seen.

For different samples the curves en u may differ strongly towards the higher values of u since only sparse observations may exceed the threshold u for large u, as shown in Figure 1.10, based on Embrechts et al. (1997). This makes the precise interpretation of en u difficult. Hill’s estimator of the tail index Hill’s estimator (1.5) is valid for a positive EVI of a heavy-tailed r.v. X and can also be constructed for dependent data. Hill’s estimator may be considered as the empirical mean excess function of the r.v. ln X for the level u = ln Xn−k . Hill’s estimator is inadequate if the underlying DF does not have a regularly varying tail, that is, F ⊆ R− does not hold,  is not positive, the sample size is not large enough, and the tail is not heavy enough ( is not big). If F ⊆ R− , then estimation by (1.5) strongly depends on the type of slowly varying function, which is usually unknown. The disadvantages of Hill’s estimate show that one has to apply several estimates of the tail index (see, Section 1.2) to deal with the complex analysis of data. Hill’s and other estimators are very sensitive to the choice of smoothing parameter. In the case of Hill’s estimator (1.5) this is the number of largest order statistics k, while for the group estimator (1.19), (1.28) it is the number m of observations in each group. The use of plots (e.g., a Hill plot or the respective plot of the group estimator) of the estimator against the smoothing parameter provides the easiest way to select such parameters.

30

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

In particular, the tail index estimates are applied to investigate the amount of finite moments of the r.v. It is well known that the pth moment exists, that is, E X1 p <  holds, if the tail index  = 1/ satisfies 0 < p <  and when the distribution has regularly varying tails, that is, belongs to class R− (Lemma 1, p. 5). A positive indicates the presence of a heavy tail. The simple tests sketched above can be successfully applied to analyze visually the features of a data attribute arising from Internet traffic measurements.

1.3.2

Analysis of Web traffic and TCP flow data

To illustrate the efficiency of the methods sketched above, two sets of real data are investigated. Data on Web traffic were gathered in the Ethernet segment of the Department of Computer Science at the University of Würzburg (Vicari, 1997). Data on transmission control protocol (TCP) flow sizes and transmission durations were measured from a mobile network. Description of the Web traffic data The measured traffic is described by a conventional hierarchical Web traffic model distinguishing a session and a page level, where the former is characteryzed by subsessions (see Figure 1.11, based on Figure 1.2 in Krieger et al., 2001). Responses to web requests are identified as the main part of the transferred data and the time between these responses is used to model the relationship between the responses. WWW session

WWW servers

WWW browser

Sub-session

Request Response

Time-out

Page

Client

Server

Figure 1.11 Hierarchical modeling of Web sessions.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

31

Consequently, the data are described at the related coarse time scales by two basic characteristics and four related r.v.s; two of these r.v.s are characteristics of sub-sessions, that is, the size of a sub-session (s.s.s.) in bytes and its duration (d.s.s.) in seconds, and two are characteristics of the transferred Web pages, that is, the size of the response (s.r.) in bytes and the inter-response time (i.r.t.) in seconds; see Table 1.3, based on Table 1 in Krieger et al. (2001). The page size data analyzed contain information on about 7480 Web pages downloaded in several TCP connections over a period of 14 days. The size of a response is defined as the sum of the sizes of all IP packets which are downloaded from a Web server to the client upon a request. To perform the analysis, we have used samples with reduced sample sizes, which were observed in a shorter period within these two weeks. The description of all these r.v.s is presented in Table 1.4, based on Krieger et al. (2001). For simplicity of the calculations, the data were scaled, that is, divided by a scale parameter s. The value of s is indicated in Table 1.4. Description of TCP flow data and research motivation We analyze real data on TCP flow traffic in an access network (Markovich and Kilpi, 2006). The data that we use are derived from a trace measured at a gateway Table 1.3 Characteristics of Web sessions. Level

Characteristic

Definition

sub-session

duration (d.s.s. [sec])

time between beginning and end of browsing a series of Web pages

size (s.s.s. [byte])

data volume of Web pages visited

inter-response time (i.r.t. [sec])

time between beginning of the old and of the new transfer of pages within a sub-session

page size (s.r. [byte])

total amount of transferred data (HTML, images, sound, )

page

Table 1.4 Description of the Web traffic data. r.v. s.s.s.(B) d.s.s.(sec) s.r.(B) i.r.t.(sec)

Sample size

Minimum

Maximum

Mean

StDev

s

373 373 7107 7107

128 2 0 6543 · 10−3

5884 · 107 9058 · 104 2052 · 107 5676 · 104

1283 · 106 1728 · 103 5395 · 104 80.908

4079 · 106 5206 · 103 4931 · 105 728.266

107 103 106 103

Reprinted from Proceedings of 1st Conference on Next Generation Internet Design and Engineering, On-line estimation of the tail index for heavy-tailed distributions with application to WWW-traffic, Markovich NM, Table 2, © 2005 IEEE. With permission from IEEE.

32

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

between a mobile network and the Internet. The important thing is that all flows have passed through a mobile access device, hence the rather limited access rates. TCP flow sizes and durations gathered from one source–destination pair both contain information about the performance of the TCP protocol at this individual level and of the network in question. A bivariate view of the flow size and flow duration provides even more information than a separate analysis of these two quantities. We assume that a user selects some random Web content, that is, the user chooses the size S according to the file size distribution, and then downloads this content using TCP. The action of a user initiates the TCP connection and its arrival time AT . In addition, the TCP protocol generates the flow departure time DT when the download is finished and, thus, determines the flow duration D = DT − AT . We aim to analyze the dependence between the r.v.s S and D. The data analyzed consist of mobile TCP connections from periods of low, average, and high network load conditions. To obtain samples as homogeneous as possible only downstream TCP flows on port 80 are considered. This means that such flows are in principle running a WWW (HTTP) application. The total number of such analyzed flows is over 610 000 and, for practical reasons, we consider here 61 disjoint bivariate samples, each of size n = 10 000. Table 1.5 (Markovich and Kilpi, 2006) states the observed ranges min max of sample means, variances, and maxima over these 61 samples. ‘Content’ in Table 1.5 refers to the size of the downloaded Web content and ‘Transmitted’ means Content plus segments retransmitted by TCP. Both are measures of the size of a flow. ‘SYN-FIN’ means from the three-way handshaking (synchronization) to finish. In other words, the flow duration is defined by the time difference between the SYN packet in the three-way handshake and the FIN packet at the end of the flow. Regarding the analysis of TCP flow data, the distribution of the maximal rate (or throughput) R = S/D and the expected throughput ER (or ES/ED) that the transport system provides are the objects of interest. A form of asymptotic independence for the pair D R is obtained in Resnick (2006, p. 239) as a result of the examination the tail of the product DR. The distributions of both S and D are heavy-tailed and their expectations may not be finite (see Table 1.9). Thus, ES/ED may be not computable. The heaviness Table 1.5 Description of the TCP flow data. Statistic

Unit

Definition

Size

kB

Content Transmitted

Duration

sec

SYN-FIN

Sample mean

Sample variance

Sample maximum

Min.

Max.

Min.

Max.

Min.

Max.

90 95

203 203

1303 1357

204553 206658

1288 1302

44522 44724

182

304

2219

52125

1074

15077

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

33

of tails means that outliers, that is, not typical observations, play a significant role in the distribution. Sometimes, this gives the wrong impression that outliers are not identically distributed with the rest of the data. The appearance of outliers in the data is quite natural. If D is very large, then S will also be quite large. But the problem is that streaming applications affect the normal behavior of TCP by prohibiting it from using the full transfer capacity available and, hence, S is much smaller than it should be. It is shown in Kilpi and Lassila (2006) that this specific data trace contains such streaming applications. Since S and D are dependent (see the results of the empirical study in Section 1.3.5) and positive, the DF of the ratio R = S/D is defined by FR x = PS/D ≤ x = =



0



  zx 0



0

fy zdydz =



0

  zx 0

FS D zx zdFD z

dFy z (1.42)

where fy z is a joint PDF of S and D, and its expectation by ER =



 0

xdFR x

if the latter integral converges. There are several alternatives to estimating FR x. The required joint bivariate distribution could be estimated by copulas and the bivariate PDF by the copula density (Nelsen, 1998). Specifically, Sklar’s theorem gives us a unique representation Fy z = C FS y FD z by means of copula Cu v if the marginal DFs FS y and FD z of two r.v.s S and D are continuous. However, how to select Cu v using statistical tools when marginal DFs are unknown remains a problem (Mikosch, 2006). One can estimate fy z by some multivariate nonparametric method, for example, the product kernel method (Scott, 1992). The problem is to find the appropriate values of bandwidths from samples of moderate size. Another alternative is to estimate the marginal DF FD z and the conditional DF FS D zx z from empirical data. FS D zx z requires a sufficiently large data set, that is, we need observations of the size for a fixed value of the duration, which may be not available. In the context of heavy-tailed distributions and statistical estimation of characteristics of heavy-tailed r.v.s, it is common practice to use the asymptotic distribution (in the sense that the sample size increases without bound) of the sample maxima as a model of the tail. We estimate a bivariate extreme value distribution (EVD) (1.48) of the pair S D instead of Fy z itself and use the EVD as its approximation to estimate fy z = 2 /yzFy z. The estimation of the EVD requires the preliminary evaluation of the marginal distributions of both S and D.

34

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

Results of the Web traffic analysis For the d.s.s., s.s.s., i.r.t., and s.r. data sets, Figures 1.12 and 1.13 show plots of Rn p against n for various p. In all cases, the values Rn p are dramatically large for large n and p ≥ 2. Hence, one may conclude that all moments of the r.v.s considered, apart from the first, are not finite. Furthermore, the plots u → en u tend to infinity for large u, implying heavy tails. These plots are close to a linear shape for all sets of data (Figures 1.14 and 1.15). The latter implies that the distributions considered can be modeled by a DF of Pareto type. The QQ plots of d.s.s., s.s.s., i.r.t. and s.r. are shown in Figures 1.16–1.19. The left-hand plots show that the exponential distribution cannot be accepted as an appropriate model for these r.v.s. The right-hand plots show that the distributions of the d.s.s., s.s.s., i.r.t. and s.r. samples are close to a generalized Pareto distribution GPD  (1.16) with different values of the parameters and  (see also Table 1.7). Figure 1.17 shows that both GPD(0.015, 1) and GPD(0.05, 0.3) could

1

1

0.6

R(p, n)

R(p, n)

0.8 0.1

0.01

0.4 0.2

1.10–3

0

50

0

100 150 200 250 300 350 400 n

0

50

100 150 200 250 300 350 n

1

1

0.1

0.1 R(p, n)

R(p, n)

Figure 1.12 Plots of Rn p against n for the duration of sub-sessions (left) and the size of sub-sessions (right) for a variety of p-values: curves corresponding to p = 05 1 2 3 4 5 are located from bottom to top, respectively.

0.01

1⋅10–3

1⋅10–3 1⋅10–4 1

0.01

10

100 n

1⋅103

1⋅104

1⋅10–4 10

1⋅103

100

1⋅104

n

Figure 1.13 Plots of Rn p against n for the inter-response times (left) and the size of responses (right) for a variety of p-values: curves corresponding to p = 05 1 2 3 4 5 are located from bottom to top, respectively.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS 2.5⋅104

1.33⋅106

4

1.32⋅106

4

1.31⋅106

1.5.10

e_n(u)

e_n(u)

2.10

1.104

1.3⋅106 1.29⋅106

5000 0

35

0

2000

4000

6000

8000

1.28⋅106

1⋅104

0

2000

4000

6000

1⋅104

8000

u

u

Figure 1.14 Exceedance en u against the threshold u for the duration of sub-sessions (left) and the size of sub-sessions (right).

3⋅104

1⋅105

e_n(u)

e_n(u)

9⋅104 2⋅104

1⋅104

8⋅104 7⋅104 6⋅104

0

0

2000

4000

6000

8000

5⋅104

1⋅104

0

2000

4000

6000

1⋅104

8000

u

u

Quantiles of GPD

Exponential quantiles

Figure 1.15 Exceedance en u against the threshold u for the inter-response times (left) and the size of responses (right).

5

0

0

2

4 d.s.s. /s

6

5

0

0

2

4 d.s.s. /s

6

Figure 1.16 QQ plots for the duration of sub-sessions (d.s.s./s) against exponential quantiles (left) and quantiles of the GPD(1, 0.3) distribution (right).

Quantiles of GPD

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

5

0

0

2

4 s.s.s./s Quantiles of GPD

Exponential quantiles

36

6

5

0

0

0.5

1 s.s.s./s

1.5

2

5

0

0

0.5

1 s.s.s./s

1.5

2

10 Quantiles of GPD

Exponential quantiles

Figure 1.17 QQ plots for the size of sub-sessions (s.s.s./s) against exponential quantiles (left) and quantiles of the GPD(0.015, 1) (top right) and GPD(0.05, 0.3) distributions (bottom right).

5

0

0

2

4 i.r.t. / s

6

5

0

0

0.5

1 i.r.t. / s

1.5

2

Figure 1.18 QQ plots for the inter-response times (i.r.t./s) against exponential quantiles (left) and quantiles of the GPD(0.015, 0.8) distribution (right).

be appropriate models of s.s.s. This implies that the QQ plot does not give a unique model to fit the underlying distribution. Table 1.6 shows the estimation of the EVI by means of the group estimator

l with the bootstrap-selected parameter m (these are denoted by lb and mb , respectively) and the selection of m from a plot (see formula (1.29)); the latter is denoted by lp . The Hill estimates with the plot- and bootstrap-selected k  ˆ pH n k and ˆ bH n k, respectively) are also presented. In order to calculate lp the averaging over m = 10 11  35, m = 10 11  40, m = 100 101  200,

Quantiles of GPD

Exponential quantiles

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

5

0

0

5

0

6 2 4 Exponential quantiles, s.r./s

37

0

0.5

1 s.r. /s

1.5

2

Figure 1.19 QQ plots for the size of responses (s.r./s) against exponential quantiles (left) and quantiles of the GPD(0.015, 1) distribution (right).

Table 1.6 Estimation of the EVI for Web traffic characteristics. r.v.

c

s.s.s.

03 04 05 03 04 05 03 04 05 03 04 05

s.r.

i.r.t.

d.s.s.

mb 8 10 22 72 71 92 42 65 156 10 13 18

p

lb

l

ˆ pH n k

ˆ bH n k

1179 0856 0902 075 087 085 069 0625 0611 0658 0539 0683

0877

0.96

0949

08

0.84

0898

0495

0.48

0712

0739

0.6

0601

Reprinted from Proceedings of 1st Conference on Next Generation Internet Design and Engineering, On-line estimation of the tail index for heavy-tailed distributions with application to WWW-traffic, Markovich NM, Table 3, © 2005 IEEE. With permission from IEEE.

m = 10 11  70 in the cases of s.s.s., d.s.s., i.r.t., s.r. is done. According to our investigation regarding the bootstrap method, the values c ∈ 03 04 05 and d = 05 were considered. For Hill’s estimate the parameters  = 2/3 and  = 1/2 were selected for the bootstrap scheme. As one can see, the values of lb are sufficiently close to the values of lp as well as ˆ pH n k and ˆ bH n k, apart from the case of i.r.t. Figures 1.20 and 1.21 illustrate the estimation of the EVI by Hill’s estimator and the group estimator l for the s.s.s., d.s.s., i.r.t., and s.r. data sets. One observes that the values of recommended by both estimates are similar. In the case of

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS 4

2

3

1.5

EVI

EVI

38

2

0.5

1

0

1

0

50

100 k, m

150

0

200

0

50

100 k, m

150

200

Figure 1.20 EVI estimation by Hill’s estimator (dotted line) and the group estimator l (solid line) for the s.s.s. (left) and d.s.s. (right) data sets. The two horizontal solid lines correspond to the levels of stability for the group and Hill’s estimators, 0.877 and 0.96 (left), 0.739 and 0.6 (right), respectively. Reprinted from Proceedings of 1st Conference on Next Generation Internet Design and Engineering, On-line estimation of the tail index for heavy-tailed distributions with application to WWW-traffic, Markovich NM, Figure 6, © 2005 IEEE. With permission from IEEE.

1.2

1.5

1 1

EVI

EVI

0.8 0.5

0.6 0

0.4

0.2

0

50

100 k, m

150

200

–0.5

0

100

200 k, m

300

400

Figure 1.21 EVI estimation by Hill’s estimator (dotted line) and the group estimator l (solid line) for the i.r.t. (left) and s.r. (right) data sets. The two horizontal solid lines correspond to the levels of stability for the group and Hill’s estimators, 0.495 and 0.48 (left), 0.8 and 0.84 (right), respectively. Reprinted from Proceedings of 1st Conference on Next Generation Internet Design and Engineering, On-line estimation of the tail index for heavytailed distributions with application to WWW-traffic, Markovich NM, Figure 7, © 2005 IEEE. With permission from IEEE.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

39

d.s.s. the difference between the values (0.739 and 0.6) arises as a result of the selection of k in Hill’s estimate corresponding to one of the stability intervals of the Hill plot; see Figure 1.20 (right). Indeed, one may select another stability interval corresponding to a value closer to 0.739. The latter example demonstrates the obvious disadvantage of the Hill plot approach, namely, the selection of k and the opportunity to use different tail index estimates. Observing the estimates of , one may conclude that the estimates of the tail index  = 1/ are always less than 2 for all data sets considered, apart from i.r.t. with 1 <  < 3. It follows from the extreme value theory (Embrechts et al., 1997), that at least the th moments,  ≥ 2, of the distribution of the s.s.s., s.r., and d.s.s. are not finite if one believes that these distributions are regularly varying. The distribution of i.r.t. may have two finite moments. It might be possible for s.s.s. (e.g., when 1 < ˆ < 2) that  < 1 and the expectation may also be infinite. According to the previous investigation regarding the confidence interval of the bootstrap estimator, one may trust c = 04 as a most cautious choice. Then ˆ < 1 holds with regard to s.s.s. and the first moment exists. The distributions of the Web traffic characteristics considered are heavy-tailed. The tail of the s.s.s. distribution is the heaviest since is the largest. With regard to the data analysis, this means that the Web data requires specific methods. More detailed information on the form of the distribution can be obtained from a sample mean excess plot or a QQ plot. Table 1.7 summaryzes the result of this preliminary analysis of the samples with our simple set of exploratory methods. Results of the TCP flow data analysis The Hill’s 

H n k, moment ˆ M n k, UH ˆ UH n k and group l estimators of the tail index were applied to observed flow sizes and durations. The results, again [min,max] ranges over all 61 samples, are given in Table 1.8 (Markovich and Kilpi, 2006). For each sample the parameter k was estimated by a bootstrapping √ method. For the group estimator we used m = l = 100 = n. The positive sign of all estimates allows us to suppose that both flow sizes and durations are heavy-tailed

Table 1.7 Comparison of the ‘rough’ methods for Web traffic data. r.v.

Number of first finite moments Rn p

Hill and group estimator

s.s.s.(B)

1

1

d.s.s.(sec) s.r.(B) i.r.t.(sec)

1 1 1

1 1 2

Type of distribution QQ plot GPD( GPD( GPD( GPD( GPD(

= 0015 = 1) = 005 = 03) = 1 = 03) = 0015 = 1) = 0015 = 08)

en u Pareto-like Pareto-like Pareto-like Pareto-like

40

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

Table 1.8 Estimation of the EVI for flow sizes (‘Content’ and ‘Transmitted’) and durations (‘SYN-FIN’). 

H n k

Content Transmitted SYN-FIN

ˆ M n k

l

Min.

Max.

Min.

Max.

Min.

Max.

Min.

05923 05760 05213

08747 11508 10034

04483 04476 03770

09794 09437 07748

05096 05239 03645

09773 09587 08562

05188 05291 03725

p = 1.5

10

0

09924 09741 08204

p =1

p = 1.0

10–1

p =2

10–1

Rn (p)

Rn (p)

Max.

0

10

10–2 10–3

ˆ UH n k

0

p = 0.5

10–2

p = 0.5

10–3

2000 4000 6000 8000 10000 n

0

2000

4000

6000

8000 10000

n

Figure 1.22 Plot of Rn p against n for the size of flows (left) and the duration of transmissions (right) for a variety of p-values: curves corresponding to p = 05 1 15 2 are located from the bottom to top, respectively.

2

EVI

EVI

1.5 1 0.5 0 0

1000

2000 k

3000

4000

1.4 1.2 1 0.8 0.6 0.4 0.2 0 0

1000

2000 k

3000

4000

Figure 1.23 estimation by Hill’s (solid line) and moment estimator (dotted line) for the flow size (left) and the duration of transmission (right). Horizontal lines show the bootstrapselected values: ˆ H n k = 0718 445 and ˆ M n k = 0840 629 (left), ˆ H n k = 0608 669 and ˆ M n k = 0683 828 (right).

distributed. All estimators, apart from the group estimator l , indicate that the flow size samples (both content and transmitted) may have infinite variance under the assumption that their distributions are regularly varying. Some samples of flow durations may have two finite first moments.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

41

2000

20000

Quantiles of GPD

Quantiles of GPD

1500

15000

10000

1000

500

5000

0

0 0

2000

4000 6000 8000 Sorted Sample

0

10000

1000

2000 3000 4000 Sorted Sample

5000

Figure 1.24 QQ plots for the size of flows (left) and the duration of transmission (right). Quantiles of GPD(1.3, 1) against the flow size (left) and of GPD(0.85, 1) against the duration of transmission (right). The linear curves correspond to the dependencies quantile against the quantile of the same distributions.

5000 1500 en(u)

en(u)

4000 3000

1000

2000 500 1000 0

0 500 1000 1500 2000 2500 3000 3500 4000 u

250 500 750 1000 1250 1500 1750 2000 u

Figure 1.25 Exceedance en u against the threshold u for the flow size (left) and the duration of transmission (right).

The number of finite moments was investigated by the statistic Rn p (Figure 1.22)9 and by the EVI estimators (Figure 1.23). The type of the distribution was investigated by QQ plots (Figure 1.24) and the mean excess function (Figure 1.25). An example of the results of this analysis for one sample of 10 000 observations is summarized in Table 1.9 (Markovich and Kilpi, 2006). The column ‘Estimators of ’ summarizes Table 1.8. The indication ‘< 1’ implies that the fractional pth moments with p < 1 may exist.

9

Figures 1.22–1.25 and 1.30–1.35 were compiled from Markovich and Kilpi (2006).

42

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

Table 1.9 Comparison of the ‘rough’ methods for TCP flow data. Number of first finite moments Rn p Content Transmitted SYN-FIN

1.3.3

1 1 0, X h ∼ c h2H−1

for large h and some H ∈ 05 1

(in this case (1.47) holds). The constant H ∈ 05 1 is called the Hurst parameter – for its estimation see Beran (1994), Willinger et al. (1995) and Kettani and Gubner (2002). The closer H is to 1, the slower is the convergence of X h to zero as h → , that is, the longer is the range of dependence in the time series. To detect long-range dependence using statistical procedures one replaces X h by the sample ACF nX h. The long-range dependence effect is typical of long time series, with several thousand points. Then one can look at lags 250 300 350, etc. If on graphing the sample heavy-tailed ACF  nX h one finds only small values, then it may be possible to model the data as i.i.d. If the sample ACF is small beyond lag q, then there is some evidence that the moving average process MA(q) may be an appropriate model (Resnick, 1997). Standard short-range dependent data sets would show a sample ACF dying after only a √ few lags and then remaining within the 95% Gaussian confidence window ±196/ n. The analysis above (see Tables 1.7, 1.9) shows that the Web and TCP flow data considered are heavy-tailed with possibly infinite variance. Therefore, the application of formula (1.44) is relevant. For a comparison we represent both estimates of the ACF for our Web data in Figures 1.26 and 1.27 and for our TCP flow data in Figure 1.28. The sample ACFs of the Web data are small in absolute value at all lags (possible exceptions are several first lags that strag outside the 95% confidence interval). The ACFs of the i.r.t. set visibly decrease after some lag. The latter tendencies are not quite so strong for the s.r. The s.s.s. and d.s.s. may be independent.

11 The process Xt is called second-order stationary if its mean EXt  does not depend on t and if the auto-covariance function depends on t and t + k only through their difference k.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

0.2

0.2

0.1

0.1

ACF

ACF

46

0 –0.1 –0.2

0 –0.1

0

200 h

–0.2

400

0

200 h

400

0

200

400

0.2 0.2

0.1

ACF

ACF

0.1 0

0 –0.1

–0.1 –0.2

0

200 h

400

–0.2

h

Figure 1.26 Estimates of sample heavy-tailed ACF (1.44) and sample ACF (1.43) for the s.s.s. (first two plots left) and d.s.s. data sets (last two plots √ right). The dotted horizontal lines indicate 95% asymptotic confidence bounds ±196/ n corresponding to the ACF of i.i.d. Gaussian r.v.s.

Both the ACFs of the TCP flow sizes are negligible at all lags apart from one. One may suppose that the TCP flow sizes are independent. The ACFs of the TCP flow durations have small values except for three lags. One can recognyze at least three clusters in the ACF plot that may indicate dependence (Mikosch, 2004). Estimates of the Hurst parameter for Web traffic and a subsample n = 1000 of the TCP flow data are presented in Table 1.10. The method proposed in Kettani

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

0.1

0.1 ACF

0.15

ACF

0.15

47

0.05

0.05

0

0

0

5000 h

–0.05

1 .104

0.2

0.2

0.1

0.1

ACF

ACF

–0.05

0

0

5000 h

1 .104

0

–0.1 0

5000 h

1 .104

–0.1 0

5000 h

1 .104

Figure 1.27 Estimates of sample heavy-tailed ACF (1.44) and sample ACF (1.43) for the i.r.t. (first two plots left) and s.r. data sets (last two plots √ right). The dotted horizontal lines indicate 95% asymptotic confidence bounds ±196/ n corresponding to the ACF of i.i.d. Gaussian r.v.s.

and Gubner (2002) under the assumption that the process is exactly second-order self-similar12 is used. It has a simple formula:

ˆ n = 05 1 + log2 1 + nX 1  H

The general investigation allows to suppose that all data sets are heavy-tailed and not LRD.

12 1 2

Xt is called exactly

second-order self-similar with Hurst parameter 0 < H < 1 if X h = The process h + 1 2H − 2 h 2H + h − 1 2H .

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS 0.1

0.1

0.05

0.05

ACF

ACF

48

0

–0.05

–0.1

0

–0.05

0

500

–0.1

1000

0

h

0.2

ACF

ACF

1000

h

0.2

0.1

0

–0.1

500

0.1

0

0

500 h

1000

–0.1

0

500 h

1000

Figure 1.28 Estimates of sample heavy-tailed ACF (1.44) and sample ACF (1.43) for one subsample n = 1000 of the TCP flow sizes (first two plots left), and durations (last two plots √ right). The dotted horizontal lines indicate 95% asymptotic confidence bounds ±196/ n.

Table 1.10 Hurst parameter estimation for Web traffic and TCP flow data. Data ˆn H

s.s.s.

d.s.s.

i.r.t.

s.r.

TCP flow size

TCP flow duration

0.493

0.488

0.508

0.507

0.498

0.506

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

1.3.4

49

Dependence detection from bivariate data

There are several ways to detect and measure the dependence among TCP flow sizes (r.v. X) and duration of transmissions (r.v. Y ). Two distribution-free measures, Kendall’s " and Spearman’s , are available. Other coefficients are given in Beirlant et al. (2004) and Weissman (2005). Importantly, all these measures can be represented by means of the so-called Pickands dependence function (Capéraà et al., 1997; Beirlant et al., 2004; Weissman, 2005). We shall briefly determine this function and apply some of its estimators to TCP flow data. Let X1  Y1   Xn  Yn  be a bivariate i.i.d. random sample with bivariate EVD Gx y. Similarly to the univariate case (Section 1.1), there exist normalizing constants ajn > 0 and bjn ∈ R, j = 1 2, such that as n → ,



P M1n − b1n /a1n ≤ x M2n − b2n /a2n ≤ y

(1.48) = F n a1n x + b1n  a2n y + b2n → Gx y

where M1n = max X1   Xn , M2n = max Y1   Yn  are the componentwise maxima (Fougères, 2004). Note that the vector M1n  M2n  will in general not be present in the original data. Let F1 x and F2 y be the DFs of X and Y , respectively. One can easily find the link between Gx y, a copula, and the dependence function: Gx y, with continuous univariate margins G1 x and G2 y,13 may be uniquely represented as a copula C, Gx y = CG1 x G2 y, by Sklar’s theorem (Nelsen, 1998). A copula is determined in terms of the Pickands dependence function At, t ∈ 0 1 (Pickands, 1981), by   log v  Cu v = PG1 X ≤ u G2 Y ≤ v = exp log uv A log uv A remarkable feature of this representation is that C depends only on Ax, but not on the margins. Hence, Gx y may be determined by margins G1 x and G2 y by the representation   log G2 y Gx y = exp log G1 xG2 y A  log G1 xG2 y see Beirlant et al. (2004). In the bivariate case the function At satisfies two properties: 1. 1 − t ∨ t ≤ At ≤ 1, t ∈ 0 1 that is, A0 = A1 = 1 and lies inside the triangle determined by points 0 1, 1 1 and 05 05; 2. At is convex.

13

Gj x, j = 1 2 is in itself a univariate extreme value DF and Fj x is in its domain of attraction (Beirlant et al., 2004, p. 254).

50

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

Cases At ≡ 1 and At = 1 − t ∨ t correspond to total independence and total dependence, respectively. Let the random pair X ∗  Y ∗  have DF Gx y. In practice, X ∗  Y ∗  are componentwise maxima over large blocks of data. It is convenient to transform initial random pairs Xi∗  Yi∗  to new pairs #i  $i  in such a way the margins are all the same.

Examples 1. The transformation #i = −1/ logG1 Xi∗ , $i = −1/ logG2 Yi∗ , leads to Fréchet distributed r.v.s # and $. 2. The transformation #i = − logG1 Xi∗ , $i = − logG2 Yi∗ , leads to exponential distributed r.v.s # and $. Pickands (1981) has shown that a bivariate DF Gx y is an extreme value DF with unit Fréchet margins if and only if    1 1 y Gx y = P# ≤ x $ ≤ y = exp − + A  x y x+y In the case of exponential margins the joint survival function of the pair # $ is given by 



y P# > x $ > y = exp − x + y A x+y





x ≥ 0 y ≥ 0

(1.49)

The latter equation is helpful in constructing the estimators of At. Many of these are presented in Beirlant et al. (2004).

Problems with estimators of A(t) 1. The estimators are not convex. They may be improved by taking a convex hull. 2. The margins G1 x and G2 x are unknown. One has to replace them by their 1 x and G 2 x, for example by empirical DFs constructed from estimates G componentwise maxima over blocks of data. The number of these maxima may be very moderate, which may lead to poor accuracy of an empirical DF. 3. The componentwise maxima may be not observable together, that is, some of them are not present in the sample. Under certain conditions one can estimate a bivariate EVD from an initial random sample X Y; see Beirlant et al. (2004, Section 9.4).

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

51

In order to approximate the FR x we can replace Fy z in (1.42) by the estimated DF of componentwise maxima   ⎞⎞ ⎛ ⎛ ˆ 2 y   log G ˆ 1 xG ˆ 2 y A ˆ⎝   ⎠⎠  (1.50) Fˆ x y ≈ exp ⎝log G ˆ ˆ log G1 xG2 y

(Beirlant et al., 2004, p. 326). We obtain ⎛ ⎛ ⎛    zx   ˆ 1 yG ˆ 2 z A ˆ⎝ FR x ≈ d ⎝exp ⎝log G 0

0

  ⎞⎞⎞ ˆ 2 z log G   ⎠⎠⎠  ˆ 1 yG ˆ 2 z log G

The simulation study provided in Hall and Tajvidi (2000) indicates that the best AHT estimators of At are  ACn t from Capéraà et al. (1997) and  n t from Hall and Tajvidi (2000):   −1 n ˆ i /# n $ˆ i /$n  1 #  AHT min   (1.51) n t = n i=1 1−t t log  ACn t =

n n n   1 1 1 log max t#ˆ i  1 − t$ˆ i − t log #ˆ i − 1 − t log $ˆ i  n i=1 n i=1 n i=1

1 Xi  and $ˆ i = − log G 2 Yi , i = 1  n, # n = n−1 n #ˆ i , $n = Here#ˆ i = − log G i=1 n n−1 i=1 $ˆ i .

1.3.5

Bivariate analysis of TCP flow data

First, we have to check the dependence between the pairs S1  D1   Sn  Dn  to apply (1.48)  and hence (1.50). For this purpose, we can calculate the ACF of the r.v.s ri = Si2 + Di2 , i = 1  n (Figure 1.29). The sample ACF of ri is small in absolute value at all lags (possible exceptions are two lags that stray outside the 95% confidence interval). One may suppose that the size–duration pairs are independent. Both A-estimators were applied to TCP flow size and duration data. The 61 j j pairs MSm  MDm  of block maxima of TCP flow size S and duration D (for block j) were used to estimate the unknown DFs G1 x and G2 x. These maxima were selected from the groups of data with similar statistical properties which correspond to the daily pattern. We also considered a larger sample of block maxima corresponding to m = 1000 points in each group. The size of the latter sample is n = 610 (Figure 1.30). j j Some pairs MSm  MDm  are not present in the initial data. These artificial pairs influence the estimation of the A-function because #i and $i are included in (1.51) with the same number i. The exclusion influences the trade-off between the bias and variance of the estimation. Here, we do not exclude these pairs from consideration

52

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

0.1

ACF

0.05

0

–0.05

–0.1

0

500

1000

h

Figure 1.29 Estimate of √ sample ACF (1.43) for ri , i = 1  n, n = 1000 with 95% confidence bounds ±196/ n.

Block Maxima of Duration(hours)

4

3

2

1

0 0

10 20 30 Block Maxima of Size (MB) j

40

j

Figure 1.30 Scatter plot of pairs of block maxima MSm  MDm , j = 1  610, when the block size is m = 1000. The pairs that are presented in the initial sample are marked by dots, whereas the pairs that are not presented (e.g., the maximal size in the group does not necessarily correspond to the maximal duration in this group) are marked by circles. Lines D = S/384 and D = S/42 indicate 384 kB/s (EDGE) and 42 kB/s (GPRS) access rates, respectively.

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

53

0.6

0.6

0.4

0.4

0.2

0.2

ACF

ACF

in order to retain a larger sample. The accuracy of the estimation is more sensitive to the number of blocks than to any other parameter. Before estimating the DFs it is important to be sure that the block maxima are independent. Then the estimation is easier. The ACFs of both maxima samples of size 61 and size 610 allow us to suppose their independence (Figures 1.31 and 1.32). One can then find parametric models of the unknown margins G1 x and G2 x and evaluate their parameters by the ML method. The ‘blocks’ method is sensitive to the size of the block. The larger the number of blocks (the smaller the size of blocks), the lower the variance and the greater the bias of the parameter estimates; some trade-off between the two is thus required. The number of blocks is connected to the dependence of the data. Roughly speaking, the size of blocks should be

0

0

-0.2

-0.2

-0.4

-0.4

-0.6

-0.6 0

10

20

30 Lag

40

50

0

60

10

20

30 Lag

40

50

60

0.2

0.2

0.1

0.1 ACF

ACF

Figure 1.31 Estimates of sample ACF (1.43) for maxima samples of size 61 corresponding to TCP flow sizes (left) and durations √ (right). The dotted horizontal lines indicate 95% asymptotic confidence bounds ±196/ n.

0 -0.1

0 -0.1

-0.2

-0.2

0

100

200

300 Lag

400

500

600

0

100

200

300 Lag

400

500

600

Figure 1.32 Estimates of sample ACF (1.43) and sample heavy-tailed ACF (1.44) for maxima samples of size 610 corresponding to TCP flow sizes (left) and durations √ (right). The dotted horizontal lines indicate 95% asymptotic confidence bounds ±196/ n.

54

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS Table 1.11 ML estimates of GEV parameters of TCP flow data Statistic

Definition

%

&

'

Size Duration

Content SYN-FIN

0332259 010263

707592 37758

460553 243327

15000

40000 Quantiles of GEV

Quantiles of GEV

12500

30000

20000

10000 7500 5000

10000 2500

10000

20000 30000 SortedSample

40000

2000 4000 6000 8000 10000 12000 14000 Sorted Sample

Figure 1.33 QQ plots of block maxima samples corresponding to TCP flow sizes (left) and durations (right).

chosen so as to give an approximately independent sample of maxima. The more dependent the data are, the larger should be the size of blocks (Leadbetter, 1983).14 Here, the GEV (1.3) is applied as such a model in a more general form: 

−1/ +  = 0 exp−1 + x−(  (1.52) H x = − x−(   exp−e 

= 0 The ML parameter estimates15 of the GEV calculated with block maxima of size 610 for both TCP samples are summaryzed in Table 1.11. We check our hypothesis regarding the GEV model with QQ plots (Figure 1.33). These evidently show not quite linear behavior. The GEV model does not accurately fit our data. Nevertheless, such a parametric model is convenient for calculating the inverse

14

The weakness of the ‘blocks’ method is its poor accuracy of approximation. The ‘runs’ method initiated by Newell (1964) seems to provide better approximation. The idea of this method is to construct blocks of unequal size by establishing a sequence of thresholds. An observation is assigned to a cluster that is bounded by two neighboring threshold values. 15 The parameters can be estimated by the method of probability-weighted moments (McNeil et al., 2005).

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS 1

0.9

0.9

0.8

0.8

A(t)

A(t)

1

55

0.7

0.7

0.6

0.6

0.5

0.5 0

0.2

0.4

0.6

0.8

1

0

t

0.2

0.4

0.6

0.8

1

t

Figure 1.34 Estimation of the Pickands dependence function by estimators  ACn t (dashed HT  line) and An t (solid line). The maxima sample of size 61 (left) and of size 610 (right). The marginal distributions G1 x and G2 x of TCP flow sizes and durations are estimated by GEV.

−1 x in formula (1.54) for bivariate quantiles. We use this −1 x and G functions G 2 1 C model to calculate the estimates  AHT n t and An t of the A-function (Figure 1.34). The convex hull of these estimates is required to further apply the quantiles formula. Both estimators are situated under the upper boundary of the triangle that indicates the dependence of TCP flow size and duration (Markovich and Kilpi, 2006). Since the size of the block maxima sample is moderate, a further improvement may be achieved by estimating the DFs G1 x and G2 x by means of the combined estimator. This estimator is a mixture of a smoothed empirical DF within the range of the sample and a parametric model (e.g., GEV) in the tail. This estimator fits the tail domain better than an empirical DF. At the same time, it may fit the DF within the range of the sample better than the GEV model, especially in the case of mixtures of distributions.  y and construct Both estimates of At allow us to get the estimate Gx bivariate quantile curves ˆ p = x y Gx ˆ QG y = p 

0 < p < 1

(1.53)

for the TCP flow data. According to the method of Beirlant et al. (2004, p. 325), ˆ ˆ 1 x = p1−w/Aw 2 x = pw/Aw one can assume that G and G for some w ∈ 0 1  in order to get Gx y = p. Then the quantile curve (Figure 1.35) consists of the points

   ˆ ˆ −1 p1−w/Aw −1 pw/Aw ˆ p = G QG  G  w ∈ 0 1  (1.54) 1 2

56

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS 20000

20000 p = 0.975 15000 p =0.95

10000

p =0.9 p =0.75

5000

0

0

10000 20000 30000 Flow Size

Flow Duration

Flow Duration

15000

10000 p = 0.95 5000

40000 50000

0

p = 0.9

p = 0.75 0

10000 20000 30000 Flow Size

40000 50000

Figure 1.35 Estimated quantile curves of TCP flow data for p ∈ 075 09 095

corresponding to estimator  ACn t: the maxima sample of size 61 (left), of size 610 (right).

Conclusions One can draw the following conclusions from the investigation: • The samples used in the analysis are of moderate size. • Size S and duration D are heavy-tailed with probably infinite second moment. • Their distributions are complicated in the sense that they do not belong to any known parametric models. • Estimates of the Pickands dependence function show that S and D are dependent. • Bivariate quantile curves show that the bivariate EVD of S D is ‘not quite heavy-tailed’ in the sense that not many observations can be considered outliers, that is fall beyond the 97.5% quantile curve. This may be a special property of these mobile TCP data. • Bivariate quantile curves are sensitive to (at least) the following: violations of the independence assumption within a block; estimation of parameters of margins of Gx y and estimates of At; and the number of componentwise maxima, or the block size.

1.4

Notes and comments

When analyzing real data, one must undertake the preliminary detection of heavy tails, as well as investigating the dependence structure of univariate data and of multivariate data. For heavy-tailed distributions and, in particular, with infinite

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

57

variance, the classical statistical methods are not adequate and flexible enough. An example is given by the sample ACF that may not represent the ACF of a heavy-tailed distribution properly. We should distinguish between methods that are valid for independent and dependent data. For example, an empirical DF cannot be applied to dependent data. The same is true regarding the ML method that is used to estimate parameters in a parametric model of the DF. The evaluation of the distributions of univariate data is particularly important for estimating multivariate quantiles and distributions. When the data are independent or weakly dependent one can apply traditional methods such as kernel estimators to estimate the PDF. The problem arises when the data are long-range dependent (Section 2.3). The exploratory techniques introduced in Section 1.3.1, apart from the ratio of the maximum to the sum, the QQ plot, and the group estimator of the tail index, do not require an i.i.d. assumption on the underlying data. The interpretation of the criteria mentioned may become hazardous when they are applied to the non-i.i.d. case. However, we have applied these tools to real data despite the sometimes evident non-i.i.d. structure. The reason is that the methods mentioned, apart from the QQ plot, have an asymptotic background and their application to samples of moderate size requires additional research. As one may conclude from Tables 1.7 and 1.9, all these methods are consistent in their conclusions. To estimate the dependence structure of two r.v.s and corresponding bivariate quantiles, one needs to evaluate the marginal DFs of maxima of both samples and the Pickands function.

1.5

Exercises

1. Generators. Generate 100 Fréchet distributed r.v.s with DF

Fx = exp − x−1/ 1x > 0 

= 15

To do this, generate 100 r.v.s Ui uniformly distributed on [0,1] and apply the transformation Xi = − ln Ui − / .

2. The ratio of the maximum to the sum. Calculate the statistic Rn p by formulas (1.37) and (1.38) for the samples X n = X1   Xn , n → . For practical calculations, one can take parts of the same sample, i.e., X i , i = 1  n. The sample X n may be generated using some random generator or real data. Plot the dependence Rn p against n for different p. Investigate this plot for large n and draw conclusions regarding the number of finite moments E X p of the distribution.

58

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

3. QQ plot Using the empirical or generated data X n = X1   Xn , construct a QQ plot, i.e., draw the dependence    n−k+1 Xk  F −1 k = 1  n  n+1 where X1 ≥ ≥ Xn are the order statistics of the sample X n = X1   Xn , and F −1 is an inverse of the DF F . Check different choices of Fx, e.g. normal, lognormal, exponential, and the generalized Pareto distribution (1.16). If the QQ plot is linear for some Fx then the underlying sample is distributed according to this Fx. Depending on whether the QQ plot is below or above the diagonal line draw conclusions whether the empirical model has a heavier tail than a parametric model or not. 4. Mean excess function. Using the empirical or generated data X n = X1   Xn , calculate the empirical mean excess function by formula (1.41). Investigate the behavior of en u for large u. For heavy-tailed distributions the function eu tends to infinity. A linear plot u → eu corresponds to a Pareto distribution, the constant 1/ corresponds to an exponential distribution, and eu tends to 0 for light-tailed distributions. 5. Estimation of the tail index. Using the empirical or generated data X n = X1   Xn , reorder the data as X1 ≤ X2 ≤ ≤ Xn  Calculate and compare the following estimates of the tail index of your data: Hill’s estimator (1.5) for some k = 1  n − 1; the ratio estimator (1.7) for some X1 < xn < Xn ; the moment estimator (1.13); the UH estimator (1.14) and the Pickands’ estimator (1.15) for some k = 1  n − 1. Investigate the sign of the estimate and draw conclusion regarding heavy tails. 6. Choice of parameter k of Hill’s estimator by a Hill plot. Considering a sample X n = X1   Xn , calculate Hill’s estimate (1.5). Plot the dependence k ˆ H n k 1 ≤ k ≤ n − 1 and then choose the estimate of ˆ H n k from an interval in which these functions demonstrate stability. Draw conclusions regarding the number of finite moments16 of the underlying distribution and the existence of heavy tails. A positive estimate

ˆ H n k may indicate heavy tails.

16

For regularly varying distributions (1.4), the moments EX  are finite only if  < 1/ (Lemma 1).

DEFINITIONS AND ROUGH DETECTION OF TAIL HEAVINESS

59

7. Repeat Exercise 6 with the group estimator (1.19), (1.28). Find an appropriate value of m from the plot m zm  m0 < m < M0 m0 > 2 M0 < n/2; see Section 1.2.4. 8. Investigation of Hill’s estimator (1.5). Generate several samples distributed with the regularly varying DFs (1.4), where ℓx = 1, ℓx = 2, ℓx = ln ln x and ℓx = ln x,17 and = 05; and with Weibull DF

1 − Fx = exp −cx1/  c = 1 = 2 c = 2 = 3 Calculate Hill’s estimate and investigate the influence of a slowly varying function ℓx on the estimate. Compare the true values of the EVI with the results of the estimation for different distributions. The estimation should be worse in the case of the Weibull distribution.

9. Repeat Exercise 8 for the group estimator. 10. Dependence detection by bivariate data. Generate 1000 r.v.s Y n = Y1   Yn such that Yi = 2Xi +1, i = 1  n. Calculate the Pickands dependence functions using the estimators  AHT n t n C and  An t (see (1.51)). In order to do this, separate both samples X and Y n into ten equal-size blocks and select the block maxima samples. Estimate the marginal DFs Gˆ 1 x and Gˆ 2 y corresponding to the block maxima of r.v.s Xi and Yi by empirical DF and by the GEV model (1.52) using the block maxima data. Estimate the parameters , (, and  of the GEV model by the ML method. C Plot  AHT n t and An t separately for two estimates of the DF. Draw conclusions regarding the dependence of X and Y .

11. Draw the bivariate quantiles curves for p ∈ 075 09 095 0975 by formula (1.54). Compare the quantile curves for both A-estimators  AHT n t C  and An t.

17

The last two ℓx require special generators which do not allow inversion of the DF. For instance, one can use the acceptance–rejection method (Law and Kelton, 2000).

2

Classical methods of probability density estimation

In this chapter the main principles of density estimation – Lebesgue’s theorem, Fisher’s scheme, L1 , L2 ,  2 approaches, the exponent method and the estimation of the PDF as solution of an ill-posed problem – are considered. The links between these approaches and examples of their application are shown. Classical methods of PDF estimation – kernel estimators, projection estimators, histogram and polygram – and their smoothing tools, among them cross-validation and the discrepancy method, are presented. We shall estimate the unknown PDF fx from a sample X n = X1      Xn  of i.i.d. observations of X, where n is the sample size.

2.1

Principles of density estimation

Lebesgue’s theorem According to Lebesgue’s theorem on densities,  fy lim dy = fx h→0 Sxh Sxh 

for almost all x, where Sxh is a closed ball with radius h and center x, and is Lebesgue’s measure. The expression under the limit can be approximated by

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

62

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

fn x =

n  1Xi ∈ Sxh  i=1

n Sxh 



This estimate was proposed in Rosenblatt (1956a) and developed in Parzen (1962). Fisher’s scheme Let fx ,  ∈ ⊆ R, x ∈ Rd , be a set of PDFs containing a true PDF fx 0  that we wish to find. Let  H0  = − ln fx fx 0 dx

(2.1)

A sequence fx n , n = X1      Xn , that leads to convergence can be found in Kullback’s metric  fx n  fx 0 dx →pn→ 0

I f fn  = H0 n  − H0 0  = − ln fx 0  Here, fx 0  is unknown. Therefore, instead of the functional (2.1) Fisher (1952) decided to minimize the empirical functional Hemp  = −

n 1 ln fXi   n i=1

constructed from the sample X n = X1      Xn  on the set fx ,  ∈ . This is called the maximum likelihood (ML) method. The consistency of the ML method is provided by the convergence    n 1 ln fXi   →pn→ − ln fx 0 fx 0 dx inf − ∈ n i=1 for any PDF fx 0 , 0 ∈ .

L1 approach The L1 approach is natural because of two remarkable effects. First of all, the estimation error in the metric space L1 is invariant for any monotone increasing, continuously differentiable, one-to-one transformation T R1 → 0 1 (the derivative of the inverse function T −1 is assumed to be continuous)1 of the data, i.e.,  1   gn x − gxdx

(2.2) fn x − fxdx = 0

1

0

The transformation Tx may be extended to T R1 → R1 .

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

63

Here, fx and gx are the PDFs of the r.v.s X and Y = TX respectively, and fn x and gn x are estimates of fx and gx. Hence, the accuracy of the estimate gn x defines the accuracy of the estimation of fx. Furthermore, according to Scheffé’s theorem, the L1 -distance between two PDFs f and fn is equal to twice the maximal deviation of the probabilities of any Borel sets (‘total variation’), calculated by PDFs f and fn :       L1 f fn  = fn x − fxdx = 2 sup  fxdx − fn xdx = 2Tf fn 

B∈

B

B

(2.3)

In practice, we are looking for different probability functions such as the DF Fx or the tail 1 − Fx, rather than for the PDF itself. Therefore, the necessity of the convergence in the metric space L1 is obvious (Barron et al., 1992). The connection between the L1 -error and Kullback’s metric is 2   fx 0  1  fx 0  ln dx = If fn  (2.4) fx 0  − fx n dx ≤ d d 2 R fx n  R

The connection between the total variation and Kullback’s metric If fn  follows from (2.3) and (2.4): 2T 2 f fn  ≤ If fn 

L2 approach The space L2 is the most convenient for constructing the estimates of the PDF and finding their accuracy with respect to the L2 -norm. ˇ Projection estimators (Cencov, 1982) are obtained by the approximation of a PDF in terms of the expansion by an orthogonal basis j t, j = 1     n:  2  n  ft − aj j t dt → min

aj

j=1

Unfortunately, there is no connection between the distances in the spaces L1 and L2 . Therefore, the results of the L2 -theory cannot be extended to L1 . For example, the convergence in the sense of the mean integrated squared error2 (MISE), cannot be extended to EL1 f fn ).

 2 approach The  2 -distance  2 f fn  = 2

See (4.12).



Rd

fx − fn x2 dx fn x

(2.5)

64

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

was introduced in Györfi et al. (1998). Since L1 f fn 2 ≤ If fn  ≤  2 f fn  2 the convergence in Kullback’s metric is stronger than the convergence in L1 , and  2 -convergence is stronger than the convergence in Kullback’s metric. The exponent method The main idea of the exponent method, proposed independently by Stratonovich ˇ (1969) and Cencov (1982), is to estimate the logarithm of the PDF, log fx, in terms of its linear expansion in some basic functions k x, e.g., polynomials, splines, trigonometric functions. Let X1      Xn be i.i.d. r.v.s with positive PDF fx bounded on some limited interval, for example 0 1. The exponent estimator is given by   m  k k x − m   (2.6) f x = f0 x exp k=1

where

m  = log



f0 x exp



m 



k k x dx

k=1

 = 1      m  ∈ Rm 

 is a normalizing multiplier to make f xdx equal to 1, and f0 x is an auxiliary PDF with the same smoothing features as fx and independent of  = 1      m  (e.g., a uniform PDF). The PDF f x may be estimated by the ML method. The uniqueness of the solution follows from the convexity of the maximum likelihood function. The maximum of the function  = ˆ − m  is taken as an estimate of , where ˆ =

m 

k=1

k ˆ k 

ˆ k =

n 1  X 

n i=1 k i

The coefficients of the expansion  are defined by the following system of equations:  f0 x exp m i=1 i i xj xdx  = ˆ j  j = 1     m

f0 x exp m i=1 i i xdx

The advantages of the exponent method are as follows: • It is simple.

• The final estimate f ˆ x is positive and integrates to 1.

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

65

• The estimate f ˆ x minimizes the Kullback–Leibler distance. • The method can be applied to heavy-tailed PDFs. However, one cannot apply the exponent method directly to a heavy-tailed PDF since the method requires the PDF fx to be bounded away from zero (fx is strongly positive) on a bounded interval. The log-density should be bounded and its derivative integrable for the convergence of a basis function expansion. If log fx belongs to Sobolev’s space W2r 0 1, r ≥ 1, that is, log fxr−1 is absolutely 2 1 continuous and 0 log fxr dx is finite, this condition holds. There are a number of problems with the exponent method: • The accuracy of the estimate depends on the choice of m. It is proved in Barron and Sheu (1991) that the rate of convergence of f x to fx in the Kullback–Leibler metric is of order On−2r/2r+1  if m ∼ n1/2r+1 and log fx belongs to Sobolev’s space. This result is extended in Koo and Kim (1996) to a wavelet basis and PDFs such as log fx belonging to the Hölder and Sobolev spaces. • The method requires a preliminary transformation of the data to a bounded interval if the PDF has long tails. • The transformation must be constructed in such a way as to avoid unbounded values of log fx in the neighborhood of the boundaries of the interval. The exponential families (2.6) arise in many problems of mathematical statistics ˇ and statistical physics. Cencov (1982)  gives the connection between m  and the Kullback–Leibler metric   = f0 x logf0 x/f xdx, under assumptions m  k xf0 xdx = 0 on the basic functions. Related works on estimates of PDFs based on exponential families include Koo and Chung (1998) and Azoury and Warmuth (2001). Density estimation from observations of the log-density Sometimes it is easier to estimate ln fx rather than fx, particularly if fx is long-tailed, for example, an exponential, Weibull or two-mode Gaussian distribution. In order to estimate ln fx one can take a sample of observations of the logarithm of the PDF and apply a regression method (Dubov, 1998). Let X n = X1  X2      Xn  be a sample of observations of some r.v. with continuous DF F x, and X1 ≤ X2 ≤    ≤ Xn be order statistics corresponding to X n .

 It is known that the square Si under the curve f x in the interval Xi  Xi+1 is a beta distributed r.v. with PDF pSi S = n 1 − Sn−1 

66

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

if 0 < S < 1 holds. Obviously, Xi+1

Si =



Xi

 f x dx = Xi+1 − Xi f zi  

 zi ∈ Xi  Xi+1

Let us construct a new sample of observations of the function ln f x at the points xiv = Xi + Xi+1 /2: R  R = xiv  ln fiv ni=1

nR = n − 1

(2.7)

where fiv =

exp   Xi+1 − Xi

 = E ln Si  = −

Since

n  1

k=1

k



E ln fiv  = ln f zi  and the variance of ln fiv is3  2 = var ln fiv  =

n  1 2 <  2 6 k=1 k

the observations R satisfy the additive model

yi =  zi  + i 



 where yi = ln fiv   zi  = ln f zi   E i  = 0 E i2 =  2  E i j / 2 = r0 for i = j r0 = 1 −  2 /6 2  Thus, one can approximate ln f x based on a new sample by some method of dependency reconstruction (Vapnik, 1982). Finally, the function ∗ ∗ x is the estimate of ln f x. x is obtained, where opt f ∗ x = exp opt The estimation of the density as a solution of an ill-posed problem The estimation of the unknown PDF using i.i.d. observations X1      Xn may be considered as an ill-posed problem of the approximate solution of Fredholm’s integral equation of the first kind,   x − tftdt = Fx (2.8) −

3

This follows from

  2 1 =

2 k 6 k=1

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

where x =



1 0

67

x ≥ 0 x < 0

The theoretical DF is unknown, but an empirical estimate of it is given, for example, in the form Fn x =

n 1  x − Xi 

n i=1

(2.9)

Equation (2.8) may be represented in operator form as Af = F

f ∈ U F ∈ V

(2.10)

where U and V are normed spaces, and A is a linear one-to-one integral operator from U to V . Definition 13 The problem of the solution of an operator equation is called correct by Hadamard if the solution exists, is unique, and is stable. The problem is ill-posed if the solution does not satisfy at least one of these three conditions ( Tikhonov and Arsenin, 1977). Regularization method In order to find the solution of (2.10) by the regularization method, one minimizes the functional R f Fn  = Af − Fn 2V + n f  in a set  of functions f from U . Here, n > 0 is a regularization parameter, and f is a stabilizing functional that satisfies the following conditions: 1. f is defined on the set . 2. f assumes real nonnegative values and is lower semi-continuous on . 3. All sets Mc = f f  ≤ c are compact in U . Vapnik and Stephanyuk (1979) proved that the application of the regularization method allows one to obtain regularized solutions fn x that converge to the true PDF fx with probability one as the sample size n goes to infinity and the regularization parameter n satisfies the following conditions: n → 0  

n=1

as n → 

exp−nn  < 

68

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

for at least one  > 0. The rate of this convergence may be obtained under additional assumptions regarding the smoothness of fx. We consider PDFs defined on the interval a b. One may suppose that fx has m continuous derivatives or that fx satisfies the Lipschitz condition ft − f ≤ Kt −  

0 < K < 

0 <  ≤ 1

t  ∈ a b

Under these assumptions the uniform convergence rate for regularized PDF estimates is proved. Theorem 2 (Stefanyuk, 1980). An asymptotic rate of convergence in the metric Ca b of the regularized estimates fn x to the true PDF fx is determined by the expression    n /21+  P lim supfn x − fx ≤ c = 1 n→ ln ln n x

if n = ln ln n /n. Here, 0 < c <  is a constant, and  depends on the smoothness of fx:  = m if fx has m continuous derivatives in the interval a b, and  =  if fx satisfies the Lipschitz condition at a b. The latter rate of the convergence is better than the rate ln n/n/1+2 proved under similar conditions in Reiss (1975) for kernel estimates (see Section 2.2.1) if the bandwidth h = ln n/n1/1+2 is used. The assignment of different norms · V and stabilizers f  allows different PDF estimators to be obtained, such as kernel, projection, histograms, and spline estimators. Theoretical background on the statistical regularization method and a numerical solution of ill-posed inverse problems can be found in Sections 7.3 and 7.5. Example 4 (Vapnik, 1982). The minimization of the regularization functional4 2  +  +  + R f Fn  = x − fd − Fn x dx + n f 2 d −

−

−

with respect to fx for a fixed regularization parameter n > 0 gives the Parzen–Rosenblatt kernel estimator (Section 2.2.1)   n

−1  x − Xi fˆh x = 2nn−1/2 K (2.11) n1/2 i=1 where Kx = exp−x.

Example 5 (Vapnik et al., 1992). Let X n = X1      Xn  be i.i.d. r.v.s with PDF fx that has compact support 0 1. In addition, it is assumed that the kth 4

In order that all integrals exist, the densities fx ∈ L2 − + are considered such that the function  + rx = − x − fd − Fn x belongs to L2 − +. These are, for instance, densities bounded away from zero just in the sets of limited measure.

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

69

derivative (k ≥ 1) of the PDF fx exists and has bounded variation on 0 1, and fx may be extended in an even manner to −1 1 and then periodically to the entire real axis so that its kth derivative will be continuous. The set of PDFs satisfying these conditions, will be denoted by ℘. According to Fikhtengol’ts (1965), any function from ℘ has a Fourier series of the form fx = 1 +

 

(2.12)

j j x

j=1

uniformly convergent to it on 0 1. Here,  1 j = 2 fxj xdx 0

j = 1 2   

j x = cosjx j = 0 1 2    , is an orthogonal basis in L2 0 1. The minimum of the regularization functional 2  1  x  1 2 R f Fn  = f k x dx ftdt − Fn x dx + n2k+2 0

0

0

(2.13)

by fx ∈ ℘ with respect to j for a fixed regularization parameter n > 0 is given by the smoothed projection estimator fˆprn x X n  = 1 +

 

(2.14)

j aj j x

j=1

−1 where j = j n  = 1 + jn 2k+2 and aj = 2/n ni=1 j Xi .

Example 6 (Markovich, 1989). Let X n = X1      Xn  be i.i.d. r.v.s with PDF fx that has compact support A B − < A B <  and is square integrable, i.e. fx ∈ L2 A B. According to Fikhtengol’ts (1965), the function fx ∈ L2 A B has a Fourier series of the form fx = a0 /2 +

   aj cos jx − /d + bj sin jx − /d 

(2.15)

j=1

uniformly convergent to fx in L2 A B. Here,  B fx cos jx − /d dx aj = 1/d A

bj = 1/d



B

A

fx sin jx − /d dx

 = A + B /2

j = 0 1 2     j = 1 2    

d = B − A /2

The minimum of the regularization functional 2   B  x R f Fn  = ftdt − Fn x dx + n A

A

B A

f 2 xdx

(2.16)

70

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

by fx with respect to aj and bj is given by the smoothed projection estimator fˆpr x = f1 x + f2 x where

(2.17)

      1 jx −  sin j f1 x = sin 0 5 + 2 d d j=1 1 + n j/d    jx −  cos j + cos  1 + n j/d2 d

√  √  1 exp x − / n − exp −x − / n

f2 x = √ √  √  2n n exp 2d/ n − exp −2d/ n     n   Xi −  Xi −  − exp − √  · exp √ n n i=1 sin j =

2.2 2.2.1

n 1 sin jXi − /d  n i=1

cos j =

n 1 cos jXi − /d

n i=1

Methods of density estimation Kernel estimators

The Parzen–Rosenblatt kernel estimator is defined by   n  x − Xi ˆfh x = 1 K  nhd i=1 h

(2.18)

where fx is defined on Rd  h > 0 is a smoothing parameter (window width or bandwidth), and Kx is a kernel function that usually satisfies the conditions  Kx ≥ 0 Kxdx = 1

(2.19)

Definition 14 A kernel Kx has an order r when a kernel function is chosen such that ⎧ ⎪ k=0 ⎨1  uk Kudu = 0 (2.20) 1 ≤ k ≤ r − 1 ⎪ ⎩ k = r

Kr−1 = 0

For r > 2 a kernel K and consequently the estimate fˆh x may have negative values. Then the estimate has to be normalized to the positive estimate:      fˆh x dx

fˆh x / +

R

+

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

71

Symmetric kernels of odd orders are not considered, since their ‘moments’  k u Kudu are always equal to 0. √ Examples of kernels are given by the normal PDF 1/ 2 exp−x2 /2, exp−x, sin2 x/x2 , and Epanechnikov’s kernel 3 Kx = 1 − x2 1x ≤ 1

4

(2.21)

Roughly speaking, the idea of kernel estimators is that all measurement points ‘are covered’ by the bell-shaped curves. The form of the ‘bell’ is determined by a kernel function. Figure 2.1 shows the bells constructed over several data points X1  X2      and the kernel estimate with bandwidth h = 0 7 for the Pareto PDF with the tail index  = 0 3. The accuracy of kernel estimates depends more on h than on Kx. The variances of all kernel estimates decay at rate Onh−1  as nh → , but the bias has order Oh2  for second-order kernels (r = 2) and order Oh4  for fourth order kernels (r = 4); see Silverman (1986). The variance of kernel estimates increases and the bias decreases as h decreases. Therefore, to select h one should find a trade-off between these effects.

2.5

2

f(x)

1.5

1

0.5

0 –0.5

0

0.5

1

1.5 x

2

2.5

3

3.5

Figure 2.1 Estimation by a standard kernel estimate with Epanechnikov’s kernel: Pareto PDF with  = 0 3 (dot-dashed line), kernel estimate with h = 0 7 (solid line), and kernel constructed over sample points (dotted line).

72

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

It is well known that the MSE for a nonvariable bandwidth kernel estimate (2.18) (d = 1) with second-order kernel5 obeys MSEfˆh  = Efˆh x − fx2

= h4 f ′′ x2 K12 /4 + nh−1 fxRK + onh−1 + h4   as h → 0, where RK = K 2 tdt, when the second derivative of the PDF is continuous. The right-hand side is minimal for   fxRK 1/5 −1/5 opt h = n

(2.22) f ′′ x2 K12 For such hopt , we obtain MSEfˆh  =

2/5 2 2/5 −4/5 5 K1 RK2 f xf ′′ x n

4

Epanechnikov’s kernel is the optimal second-order kernel for the estimate fˆh in the sense of the least K1 RK2 . But an ideal estimate cannot be attained since hopt depends on the unknown derivative f ′′ y. In practice, fy can be replaced by the preliminary kernel estimate   n  x − Xi ˆfh x = 1 K

1 nh1 i=1 h1

Then the derivative f ′′ x is evaluated by the second derivative of fˆh1 x. Let us choose a kernel Kx having two bounded derivatives and satisfying (2.19) (the nonnegativity of the kernel is not required), and (2.20), for example, the Epanechnikov kernel. Hence, since f ′′ x is assumed to be continuous, fˆh′′1 x is the estimate of f ′′ x; see Prakasa

√ Rao (1983). 1/13 The smoothing parameter h1 of fˆh1 x can be taken to be X 84 /5n2  (X is the standard deviation of the sample X n ), which is consistent with the parameter of the kernel estimate with Gaussian kernel guaranteeing optimal estimation of f ′′ x2 dx (Wand et al., 1991). It can be chosen by data-dependent methods such as cross-validation (Chow et al., 1983) or by the discrepancy method (Markovich, 1989). A kernel estimator has the following advantages: • It can be easily applied to multivariate densities. • Recursive behavior: it is easy to use it for on-line estimation. It has the following disadvantages: • It may be negative for kernels of order r > 2 . 5

The simplest example of such kernels is provided by a symmetric kernel with compact support.

CLASSICAL METHODS OF PROBABILITY DENSITY ESTIMATION

73

• It may exhibit boundary effects (distortions) on compactly supported PDFs due to kernel truncation near the boundaries. • Kernel estimation is not good for nonsmooth PDFs, such as a uniform PDF. Consistency conditions The asymptotic conditions for the convergence of a kernel estimate to the true uniform continuous PDF were obtained in Parzen (1962) and Nadaraya (1965). According to Parzen (1962) for a marginal distribution and Murthy (1966) for a multivariate distribution, h→0

as n → 

(2.23)

provides an asymptotically unbiased estimate when Kx obeys (2.19) and sup− 0 satisfy . Roughly speaking, f˜n x is a sum of a pi = 1 and bi n = Ni−1 0. We wish to estimate the whole PDF fx using a sample X n of moderate size n. For this purpose, we employ here the idea of a separate estimation of the ‘tail’ and ‘body’ of the PDF. Such a separation is reasonable since the risks related to the estimation of the tail and the body of the PDF have qualitatively different values. We consider the combined estimate  N t ∈ 0 Xn−k  ˜f t  N  = f t (3.1) f t t ∈ Xn−k   Here Xn−k is some r.v. defined subsequently and f N t is some nonparametric estimate of ft on the interval 0 Xn−k . The latter is represented by a finite series expansion in terms of trigonometric functions, for example, k t = 4/ 1/2 cos 2k − 1 /2t t ∈ 0 1 k = 1 2    :   N t 1  N f t =   (3.2) Xn−k j=1 j j Xn−k Here N is a smoothing parameter determining the complexity of the estimate. Let f t = 1/t−1/−1 + 2/t−2/−1

(3.3)

102

HEAVY-TAILED DENSITY ESTIMATION

be an estimate of the ‘tail’ of the PDF ft. The estimate f N t is calculated by that part of the sample located on the interval 0 Xn−k  and a tail estimate f t is calculated by the rest of the sample.  The estimate (3.1) may not be a real PDF, that is, 0 f˜ t  N  dt = 1 and f˜ t  N  < 0 for some t. In this case, one can take ⎧ f˜ t  N  ⎨  t ∈ A  ∗ ˜ t  N 1f˜ t  N  > 0dt f t  N = (3.4) f 0 ⎩ 0 t ∈ A

 for A = t ∈ 0   f˜ t  N  > 0 instead of f˜ t  N  to provide  ∗ f t  N dt = 1 and f ∗ t  N  ≥ 0. If 0 Xn−k f N tdt = 1 (3.5) 0

holds, then



 0

−1/ −2/ f˜ t  N dt = 1 + Xn−k + Xn−k

(3.6)

follows. The EVI  is the most important parameter to describe the shape of the tail (see Section 1.2). In Hill’s estimator and many others (see Section 1.2.3) the parameter  is calculated by the k + 1 largest values of the order statistics X1 ≤    ≤ Xn of the sample X n . The parameter k indicates Xn−k and, therefore, the part of the distribution which controls the extreme values of the underlying r.v. It can be estimated by different methods described in Section 1.2, e.g., by the bootstrap technique. One has first to estimate k to fit the ‘tail’ and adapt then the ‘body’ of the PDF as described in Section 3.2.1. Remark 1 The boundary between two parts f N t and f t requires additional smoothing to avoid a gap at the point Xn−k . This can be done, for instance, by means of a kernel that has a special boundary property linking it with f Xn−k  if a kernel estimator is used as f N t. One

can propose joint conditions 

for estimate (3.2)  as a slope line between two points Xn−k−1  f N Xn−k−1  and Xn−k  f Xn−k  , namely, ⎧ ⎪ f N t t ∈ 0 Xn−k−1  ⎪ ⎪ N ⎨ f X

  − f X   n−k n−k−1 t − Xn−k−1 + f N Xn−k−1  t ∈ Xn−k−1  Xn−k  f˜ t  N  = ⎪ Xn−k − Xn−k−1 ⎪ ⎪ ⎩ f t t ∈ Xn−k  

The combined estimate has the disadvantage that it is necessary to select an appropriate tail model. It has two advantages: First, having selected an appropriate tail model, the ‘tail’ of the PDF can be accurately evaluated (Figure 3.3 shows an example of the poor selection of the tail model). Second, one is free to select many combinations of nonparametric and parametric estimator.

HEAVY-TAILED DENSITY ESTIMATION

3.2.1

103

Nonparametric estimation of the density by structural risk minimization

To estimate the PDF on the interval 0 Xn−k , we shall use here a technique for dependence reconstruction (Vapnik and Stefanyuk, 1979; Stefanyuk, 1984). In the following, let n∗ = n − k. We suppose, without loss of generality, that the unknown PDF ft is continuous and located on 0 1. For this purpose, we transform that part of a sample X n located on the interval 0 Xn−k  to 0 1 by the formula: tn−k−i+1 =

Xi Xn−k



i = 1     n − k

Constructed from the given data X n = X1      Xn , the empirical DF (2.9) is calculated at some independent uniformly distributed points 1      l , i.e. i = i/l + 1 i = 1     l. Hence, we get the data 1  y1      l  yl  , where yi = Fn i  = Fi  + i is assumed, that is, the values of the empirical DF are considered as the values Fi  of the original DF corrupted by the noise i . Since the values yi are correlated, the noise is correlated, too. The empirical DF is an unbiased estimate of Ft, therefore Ei = 0. The variance of the noise depends on the value of Ft, that is, it changes from one point to the next. Considering an optimal adaptation to the data in this case, it is recommended by the theory of the least-squares method to minimize the empirical risk  − GT  − G = y − FT R−1 y y − F

(3.7)

instead of carrying out a minimization of the form y − FT y − F. Here Ry is the covariance matrix of the vector y = y1      yl T , where  = By, G = T BF, BT B = R−1 y and F = F1       Fl   . The matrix B transforms the correlated observations yi into the uncorrelated i with variance equal to 1.2 All asymptotic properties of the least-squares method, unbiasedness and minimal variance of the linear, unbiased estimates, are preserved. The estimation of the PDF from data with correlated or independent noise is different. If we are dealing with independent data, increasing the sample size gives us more accurate estimates. If we use correlated data, then increasing the number of observations is subject to diminishing returns – the correlated points may ‘repeat’ and fail to provide any new information. The use of the covariance matrix takes into account the co-location of different parts of the distribution – the structure of the PDF – and helps to estimate multimodal PDFs (PDFs of mixtures of distributions). ˇ The idea of using correlated data is essentially used in a variety of methods (Cencov, 1982; Dubov, 1998; Kooperberg et al., 1994; Vapnik, 1982).

2

The asymptotical normality of i may be proved. Uncorrelated, normally distributed i are independent. The independence is required for the further application of the structural risk-minimization method.

104

HEAVY-TAILED DENSITY ESTIMATION

Following this idea of transforming the correlated observations i  yi  i = 1     l , we will now determine the estimate g N t =

N 

j j t

j=1

of the ‘body’ of the PDF by an application of the structural risk-minimization see Vapnik and Stefanyuk (1979) and Vapnik (1982). Let G  = method;  N g tdt be the corresponding DF estimate and  = 1      N T . 0 The original structural risk-minimization method requires independent measurements i  yi . The idea of this method is to formulate the optimal estimation by means of the given data as minimization of the mean risk y − G 2 dP y → min 

where the measure P  is unknown and  =  N  is the vector of the parameters. This task is performed by the minimization of its upper bound J = gN l

l 1 y − Gi  2  l i=1 i

(3.8)

where the penalty function gN l depends on the Vapnik–Chervonenkis dimension and may have different forms for different classes of models. Such types of bounds follow from fundamental estimates of the deviation of the mean risk from its empirical analogue (for further details, see Vapnik, 1982). Following the arguments leading to (3.7) in the case of correlated points, instead of J in (3.8) the minimization of   l−1 Y − FT R−1 y Y − F JN  = (3.9)  1 − l−1 N + 11 + ln l − lnN + 1 − ln   with respect to N and  is used, where  > 0 is a confidence level,  z z > 0

z =  z ≤ 0

and Y = Y1      Yl T 

Yi = y i −



i 0

1 t dt 1

By the choice of an optimal complexity N for a given number l of points i the structural risk-minimization method selects the values of these parameters which provide a lower minimum of the mean risk than those parameters corresponding to the minimum of the empirical risk.

HEAVY-TAILED DENSITY ESTIMATION

105

As yi one can take the estimate n∗ t of the unknown DF Ft at i determined below in (3.11). Furthermore, we use:   i  N 

 j   T j  j t − 1 t dt F = F1      Fl  Fi = 1 0 j=2 F = A · 1 

Here the elements of the l × N − 1 matrix A are given by  i  j j t − 1 t dt Aij = 1 0 1 j tdt i = 1     l j = 2     N j = 0

1 is the N − 1 × 1 vector of parameters j  j = 2     N . The matrix ⎞ ⎛ r 1 1 0       0  ⎟ ⎜ ⎜1 r2 2 0     ⎟ ⎜ ⎟ ⎜      ⎟ ⎜    ⎟ ⎜0   ⎟ R−1 y =⎜   ⎟ ⎜             0 ⎟ ⎜ ⎟ ⎜ ⎟ ⎝     0  r  ⎠ l−2

l−1

(3.10)

l−1

0       0 l−1 rl

with

r1 = ri−1 =

n∗ F2   F1  F2  − F1 

rl =

n∗ 1 − Fl−1  1 − Fl  Fl  − Fl−1 

n∗ Fi  − Fi−2   Fi  − Fi−1  Fi−1  − Fi−2 

i = −

n∗  Fi+1  − Fi 

i = 3 4     l

i = 1 2     l − 1

is used. However, the following estimate DF Ft: ⎧   1 t ⎪ ⎪  ⎪ ⎪ ∗ ⎪ 2n t1 ⎪ ⎪ ⎪   ⎨ m − 05 1 t − tm + ∗  n∗ t = ⎪ n∗ n tm+1 − tm ⎪ ⎪ ⎪ ∗   ⎪ ⎪ t − tn∗ ⎪ ⎪ n − 05 + 1  ⎩ n∗ 2n∗ 1 − tn∗

n∗ t is used instead of the unknown 0 < t ≤ t1 tm < t ≤ tm+1  t n∗ < t ≤ 1

m = 1     n∗ − 1

(3.11)

106

HEAVY-TAILED DENSITY ESTIMATION

The minimization algorithm has two stages: 1. The DF Ft and R−1 y are estimated from the sample using (3.11) and (3.10). is replaced by its estimate and the parameters of the PDF 2. In (3.9) R−1 y estimate g N t are obtained by the minimization of J N  with respect to N and . 1 The method preserves 0 Nj=1 j j tdt = 1.

Computational notes 1. Let  = 005.

2. Stefanyuk (1984) recommended selecting l = 5n/ ln n to provide the asymptotic minimum of the L2 error as n → .

3. To avoid division by zero in the formula (3.11) of the estimate n∗ t of the empirical DF, the points tm  m = 1     n∗ cannot repeat each other.3 4. 1 is calculated by

1 =

1−

N

j=2 j j

1



5. One minimizes the empirical risk l−1 Y − A1 T R−1 y Y − A1  over 1 = T 2      N  for each fixed N . The minimum gives the following estimate:

−1 T −1 ∗N = AT R−1 A Ry Y (3.12) y A Among the vectors ∗N  N = 2 3     Nmax (where Nmax is the maximum value of N considered) one selects those corresponding to the minimum of JN .

6. The empirical risk (the numerator of (3.9)) has to decrease with increasing N . If this risk increases, then the matrix of the system is nearly singular. 7. The minimum of (3.9) is not necessarily reached for a maximal N . For such N the empirical risk is minimal, but the inverse denominator of (3.9) is maximal. 8. Usually, 2 ≤ N ≤ 20.

9. Finally, the ‘body’ estimate of the PDF is calculated by the formula   N x 1  N f x =   Xn−k j=1 j j Xn−k with x ∈ 0 Xn−k .

3

For continuous Fx, repetitions are impossible.

HEAVY-TAILED DENSITY ESTIMATION

107

10. One can use another complete system of basis functions k t k = 1 2     instead of the trigonometric functions. Remark 2 Kernel estimates typically do not fit all modes of a multimodal PDF well enough. Usually, a kernel estimate over-smoothes one mode and fits another well, if, for instance, a mixture of two normal distributions is considered. Vapnik and Stefanyuk (1979) found that the approach considered works better than kernel estimates for the estimation of multimodal PDFs (see Figure 3.3).

3.2.2

Illustrative examples

To demonstrate the power of the combined estimator ⎧N t ∈ 0 Xn−k  ⎪ j=1 j j t ⎪ ⎪ ⎪ ⎪ N ⎪  ⎪ ⎨ f Xn−k  − f Xn−k−1  t − X n−k−1 Xn−k − Xn−k−1 f˜ t  N  = ⎪ ⎪ ⎪ t ∈ Xn−k−1  Xn−k  +f N Xn−k−1  ⎪ ⎪ ⎪ ⎪ ⎩ 1/t−1/−1 + 2/t−2/−1  t ∈ Xn−k   (3.13)

and its ability to estimate long-tailed PDFs and their mixtures, some illustrative examples, motivated by the measurements in Bolotin et al. (1999) and Roppel (1999), are presented. For this purpose we have generated samples which follow a mixture of two distributions, i.e. the PDF is determined by fx = 05f1 x + 05f2 x Particularly, we consider mixtures of • a Burr distribution, Burr  = Burr08 −2, with PDF f1 x = x−1 1 + x −−1  and  =

1 , 

x > 0  > 0  > 0

 = − 1 , and a gamma distribution Ga = Ga9 with PDF f2 x =

x−1 exp−x  

x > 0  > 0

(see Figure 3.1); • a Gamma distribution Ga(2.5) as F1 x with f1 x = distribution with PDF  +1 k  f2 x = k k+x and k = 1  = 03 (see Figure 3.2);

dF1 x dx

• the Gamma distributions Ga(1.9) and Ga(10) (see Figure 3.3).

and a Pareto

108

HEAVY-TAILED DENSITY ESTIMATION 1

1

0.1 f(x)

f(x)

0.5

0

–0.5

0.01 1.10–3

0

2

4 x

6

1.10–4

8

0

10

20 x

30

40

Figure 3.1 Estimation of the PDF of a mixture of a gamma and a Burr distribution (dotted line) by the combined estimate (3.13) (solid line) and the kernel estimate with Epanechnikov’s kernel (dashed line): ‘body’ reconstruction (left); ‘tail’ reconstruction (right). The bandwidth is selected by the over-smoothing method (2.30).

1

0.2

0.1

f(x)

f(x)

0.3

0.01

0.1

0

0

5

10 x

15

1.10–3

20

0

10

20 x

30

40

Figure 3.2 Estimation of the PDF of a mixture of gamma and a Pareto distribution (dotted line) by the combined estimate (3.13) (solid line): ‘body’ reconstruction (left); ‘tail’ reconstruction (right). The over-smoothing bandwidth is given by h = 1279 × 107 . The kernel estimate is over-smoothed and not presented.

0.3

f(x)

f(x)

0.2

0.1

0 0

2

4 x

6

8

1 0.1 0.01 1. 10–3 . 1 10–4 1. 10–5 1. 10–6 1. 10–7 1. 10–8 1. 10–9

0

10

20

30

40

x

Figure 3.3 Estimation of the PDF of a mixture of two gamma distributions (dotted line) by combined estimate (3.13) (solid line) and the kernel estimate with Epanechnikov’s kernel (dashed line): ‘body’ reconstruction (left); ‘tail’ reconstruction (right). The bandwidth is selected by the over-smoothing method (2.30). The tail model is wrongly selected.

HEAVY-TAILED DENSITY ESTIMATION

109

Note that the first two mixtures are heavy-tailed due to the presence of Burr and Pareto distributions, and the last mixture is light-tailed. In all cases the sample size n = 200 has been used. The bootstrap values k1 ∈ 5 12 4 and k ∈ 29 68 23 (see formulas (1.10)–(1.12)) in Hill’s estimator and the number of terms N ∈ 18 4 11 in expansion (3.2) corresponding to the minimum of J N , have been selected for the mixtures mentioned. Typical mistakes 1. Sometimes the mixtures look like unimodal distributions (Figure 3.2). Therefore one tries to find an appropriate parametric model among wellknown distributions, which is difficult. 2. Figure 3.3 demonstrates the wrong selection of the tail model and the estimator of the tail index. The gamma tail is lighter than the Pareto tail (3.3) that is used in (3.13). The Hill estimator cannot be applied here since the tail index of the light-tailed mixture of gamma distributions is negative. Due to these mistakes, the gap between the nonparametric and the parametric part is visible. This example demonstrates that a rough investigation of the heaviness of tails (Section 1.3.1) is necessary before estimation. 3. The kernel estimate with compactly supported Epanechnikov kernel (i.e., the kernel is defined on a finite interval) is truncated beyond the range of the sample and cannot be used to estimate the tail. In order to estimate the tail of the PDF better a specific transformation of the data is proposed (see Section 4.3).

3.2.3

Web data analysis by a combined parametric–nonparametric method

We apply the combined estimator (3.1) to four Web data characteristics described in Section 1.3.2. To simplify the calculations, the data were scaled, that is, the values were divided by the scaling parameter s (see Table 1.4). The values of the parameters of the combined estimate are presented for each r.v. in Table 3.1. Here, k1 provides the minimum value of (1.12), and k is calculated by (1.10) and (1.11). In these formulas we take  = 2/3,  = 1/2. The corresponding order statistic Xn−k , which is, roughly speaking, the ‘boundary’ between the ‘tail’ and the ‘body’ of the PDF (see (3.1)) as well as Hill’s estimate of the tail index   H n k are also presented. The number of terms N in the expansion (3.2) provides the minimum of the functional (3.9). The vectors of the coefficients  in (3.2) calculated by (3.12) are given in Table 3.2. The combined estimates are presented in Figures 3.4–3.7. Each figure consists of two graphs to demonstrate better the behavior on the ‘tails’ and ‘bodies’ of the corresponding PDFs. The scaled values x/s are presented on the x-axis. Calculated by the formula fx = 1/sgx/s, the values of the PDF estimates are presented on

110

HEAVY-TAILED DENSITY ESTIMATION

Table 3.1 Parameters of the combined estimate. r.v.

k1

k

Xn−k /s

s.s.s. d.s.s. s.r. i.r.t.

6 4 4 9

42 28 31 70

0214 4071 0108 0109

  H l k 0952 06 0615 1001

N

s

15 12 8 4

107 103 106 103

Reprinted from Computer Networks, 40(3), pp. 459–474, The estimation of heavy-tailed probability density functions, their mixtures and quantiles, Markovitch NM and Krieger UR, Table 4, © 2002 Elsevier. With permission from Elsevier.

Table 3.2 Vectors of optimal coefficients. r.v.



s.s.s.

(1.584, 0.929, 0.688, 0.384, 0.365, 0.297, 0.373, 0.381, 0.365, 0.289, 0.225, 0.186, 0.263, 0.159, 0.153)T (1.542, 0.635, 0.405, 0.175, 0.108, 0.172, 0.174, 0.157, 0.173, 0.125, 0.219, 0.189)T (1.621, 1.079, 0.862, 0.590, 0.486, 0.329, 0.300, 0.076)T (1.561, 0.763, 0.512, 0.121)T

d.s.s. s.r. i.r.t.

Reprinted from Computer Networks, 40(3), pp. 459–474, The estimation of heavy-tailed probability density functions, their mixtures and quantiles, Markovitch NM and Krieger UR, Table 5, © 2002 Elseiver. With permission from Elseiver.

1.10–16

3.10–6

1.10–17 f(x)

f(x)

2.10–6

1.10–6

1.10–18 1.10–19 1.10–20

0 0.01

0.1

1 x/s

10

1.10–21

0

500 x/s

1000

Figure 3.4 Estimation of the PDF of the sub-session size by the combined estimate. Reprinted from Computer Networks, 40(3), pp. 459–474, The estimation of heavy-tailed probability density functions, their mixtures and quantiles, Markovitch NM and Krieger UR, Figure 4, © 2002 Elsevier. With permission from Elsevier.

HEAVY-TAILED DENSITY ESTIMATION

111

1.10–10 0.0015 1.10–11 1.10–12

f(x)

f(x)

0.001

1.10–13 1.10–14

5.10–4

1.10–15 0 0.1

1 x/s

1.10–16

10

0

500 x/s

1000

Figure 3.5 Estimation of the PDF of the duration of the sub-session by the combined estimate. Reprinted from Computer Networks, 40(3), pp. 459–474, The estimation of heavytailed probability density functions, their mixtures and quantiles, Markovitch NM and Krieger UR, Figure 5, © 2002 Elsevier. With permission from Elsevier. 1.10–18

4.10–5

1.10–19

f(x)

f(x)

1.10–20 2.10–5

1.10–21 1.10–22 1.10–23

0 0.01

1.10–24 0.1

1

10

0

x/s

500 x/s

1000

Figure 3.6 Estimation of the PDF of the response size by the combined estimate. Reprinted from Computer Networks, 40(3), pp. 459–474, The estimation of heavy-tailed probability density functions, their mixtures and quantiles, Markovitch NM and Krieger UR, Figure 6, © 2002 Elsevier. With permission form Elsevier.

the y-axis. Here, gy is the estimate (3.1) resulting from the scaled data Yi = Xi /s, where Xi  i = 1     n , are empirical measurements. Since   H n k > 0 for all r.v.s considered, one may conclude that their distributions have heavy tails. However, d.s.s. and s.r. have larger 1/ and, hence, heavier tails, than s.s.s. and i.r.t.

3.3

Barron’s estimator and  2 -optimality

A similar approach to a combined parametric–nonparametric method is realized in Barron et al. (1992). Specifically, let Pn = An1      Anmn be partitions of the real line 0  into finite intervals (bins) by quantiles G−1 j/mn  1 ≤ j ≤ mn − 1,

112

HEAVY-TAILED DENSITY ESTIMATION 1.10–7

0.03

1.10–8 f(x)

f(x)

0.02

0.01

1.10–9 1.10–10 1.10–11

0 0.01

0.1

1.10–12

1

500

0

1000

x/s

x/s

Figure 3.7 Estimation of the PDF of the inter-response time by the combined estimate. Reprinted from Computer Networks, 40(3), pp. 459–474, The estimation of heavy-tailed probability density functions, their mixtures and quantiles, Markovitch NM and Krieger UR, Figure 7, © 2002 Elsevier. With permission form Elsevier.

  of an arbitrary distribution Gx j = Anj dFn x = 1/n ni=1 1Anj Xi  Fn x an empirical DF, n the sample size. The estimator is defined as follows: fˆB x = gx

1/n + j  1/n + 1/mn

x ∈ Anj 

1 ≤ j ≤ mn 

(3.14)

The consistency of the estimate is provided by the conditions mn →  and mn /n → 0 as n → . The behavior of the DF beyond the range of the sample, as x > Xn , is unknown. Therefore, one has to use the asymptotic models of the DF, which follow from the extreme value theory (Gnedenko, 1943). In the estimate fˆB x, different parametric models are selected as DF Gx (e.g., lognormal, normal, Weibull distributions). The parameters of these models may be estimated by the ML or moment methods. Indeed, the choice of gx = G′ x has a strong effect on the estimate of the DF in the tail domain, when x ∈ Anmn = G−1 mn − 1/mn   (this is the area of sparse observations), since 1/n + mn 1/n + mn  gxdx = 1 − Gx Fˆ x = 1/n + 1/mn x 1/n + 1/mn

holds. In Berlinet et al. (1998) the consistency of fˆB x in a sense of the  2 -distance under some assumptions on the PDF is proved. Two problems related to an optimal choice of partitions and the auxiliary function gx are solved in Vajda and van der Meulen (2001). Following Györfi et al. (1998), upper bounds of the mean  2 -distance (2.5) over some classes of PDFs are minimized to find an optimal mn . By the way, a similar estimate fˆs x =

mn   + i  i=1  + 1/mn

x ∈ Ani 

with the separation of the domain of definition domain of PDF fx into equal partitions has been obtained by a regularization method in Stefanyuk and Karandeev

HEAVY-TAILED DENSITY ESTIMATION

113

(1996). However, this estimate is intended for finite PDFs. A Bayesian approach to the choice of parameters, namely, the smoothing parameter  and the number of intervals mn , has been used. This approach provides the minimum of the MISE for known prior distributions (Stefanyuk and Karandeev, 1996). Barron’s estimator reffers from two disadvantages. First, it is necessary to select an appropriate tail model. Second, the tail model gx distorts the estimate of the ‘body’ of the PDF for samples of moderate size. This influence becomes weaker as the sample size increases (K˙us and Vajda, 1996).

3.4

Kernel estimators with variable bandwidth

If Fx is heavy-tailed with PDF fx, the well-known PDF estimators such as the histogram and the kernel estimator perform quite poorly. Since the smoothing parameter (e.g., the bandwidth h of a kernel estimator) is fixed across the entire sample, these estimators may provide a misleading estimation in the tail domain or over-smooth the body of the PDF. Some examples are given for suicide data by Silverman (1986, p.18) and in Figures 3.3 and 3.8. To overcome this problem it is natural to use kernel estimators with kernels that vary from one point to another – the so-called variable bandwidth kernel estimator (Abramson, 1982; Devroye and Györfi, 1985; Hall and Marron, 1988; Hall, 1992) fˆ A x h = nh−1

n  i=1

 fXi 1/2 K x − Xi fXi 1/2 /h 

Since fXi  is unknown, the estimator  f A x h1  h = nh−1

n  i=1

 fˆh1 Xi 1/2 K x − Xi fˆh1 Xi 1/2 /h

(3.15)

f(x)

2

1

0

0

1

2

x

3

4

5

Figure 3.8 Kernel estimates with Epanechnikov’s kernel and different bandwidth values h for a Fréchet PDF with shape parameter  = 15 (solid line): h = 005 (dotted line), h = 1 (dot-dashed line).

114

HEAVY-TAILED DENSITY ESTIMATION

is used in practice. Usually, the nonvariable bandwidth kernel estimator (2.18) is used as a pilot estimator fˆh x. The variable bandwidth kernel estimator fˆ A x h provides the mean squared error MSEfˆ A x = Efˆ A x h − fx2 (3.16)   2  2  4 K3 d 1 c = h8 + +onh−1 + h8 + fx3/2 24 dx fx nh as h → 0 uniformly in x ∈ R, if fx has four continuous derivatives and is bounded away from zero on R ≡ x ∈ R  for  some y ∈ R x − y ≤  (Silverman, 1986; Hall and Marron, 1988). Here, c = K 2 tdt and K3 is determined by Definition 14. Hence, the fastest achievable order n−8/9 of the MSE is attained if fx has four continuous derivatives, the kernel function Kx satisfies t4 Kt dt <  Kxdx = 1 sup Kx <  (3.17) x

and h = c1 n for some constant c1 . For a nonvariable kernel estimator this improves to n−4/5 (Hall and Marron, 1988). Since the variance of any kernel estimate has rate O1/nh, as nh → , then the reduction of MSE arises by the reduction of the bias E fˆh x − fx which has order h4 for the variable bandwidth kernel estimates. It is proved by Hall and Marron (1988) for the estimator (3.15) used in practice that −1/9

 f A x h1  h = fˆ A x h + cZnh−1/2 + onh−1/2 

(3.18)

where c is a constant, Z is a standard normal r.v., and h1 ≃ n−1/5 . The value c = ch1  may be obtained from Hall and Marron’s formula (4.5) and the application of Lindeberg’s theorem to  f A x h1  h − fˆ A x h, which is a sum of i.i.d. r.v.s (Petrov, 1975). Then the bias of  f A x h1  h is the same as for fˆ A x h. The variance of A  f x h1  h is defined by   var  f A x h1  h = var fˆ A x h + c2 nh−1 + onh−1  (3.19) under the assumption EZ · fˆ A x h = 0. The variance of  f A x h1  h is a little A larger than the variance of fˆ x h. However, both are of the same order of magnitude. The modified estimator   n x − Xi fXi 1/2 1  1 x − Xi ≤ 1 + x  log h−1 fXi 1/2 K fˆ H x h = nh i=1 h

is presented in Hall (1992). This modification prevents very large Xi s, that is, Xi s a long way from x. The bias of fˆ H x h has the same rate h4 .

HEAVY-TAILED DENSITY ESTIMATION

115

It is remarkable that none of the estimators mentioned require the assumption that Kx is nonnegative, because the fourth order (see Definition 14, p. 70) of Kx (and thus, possible negativity) is not required. This implies that variable bandwidth kernel estimates have the fastest rate of convergence without the disadvantage of negativity. For nonvariable kernel estimates this rate can only be achieved using  fourth-order kernels (such as t2 Ktdt = 0) which take negative values and, lead to negative kernel estimates (Silverman, 1986). It is argued in Hall (1992) that the MISE is not a convenient measure of quality 2  A for the estimate fˆ A x h, since the asymptotic error E fˆ x h − fx dx – or, more precisely, the variance of the estimate – is driven by the tail behavior of fx. As the tails of the distribution become lighter, the MISE converges to that of an nonvariable kernel estimate, n−4/5 . Another important problem is the smoothing or selection of the bandwidth h in kernel estimators. We find that MSEfˆ A x is minimal on h

opt

=



F2 8F1

1/9

n−1/9 

(3.20)

where  F1 = K3 /242 d/dx4 1/fx

2



F2 = cfx3/2 

(3.21)

For such hopt ∼ n−1/9 it evidently follows that MISE ∼ n−8/9 . Indeed, the parameter hopt depends on the unknown derivative d/dx4 1/fx. The estimation of derivatives of the PDF is a complicated problem in itself. The estimation of an additional derivative is more difficult than estimating an additional dimension. For example, the optimal asymptotic mean integrated squared error (AMISE) rate for the second derivative is On−4/9  which is the same (slower) rate as for the optimal AMISE of a five-dimensional multivariate frequency polygon PDF estimator (Scott, 1992, p. 132). Hence, for practical computation the data-dependent methods (e.g., cross-validation or the discrepancy method) for the selection of h may be better if one is dealing with samples of moderate size. The cross-validation method produces consistent nonvariable kernel estimates in the L1 metric in the case of a distribution with bounded support (Chow et al., 1983). For heavy-tailed PDFs cross-validated estimates do not converge since h → 0 as n →  (Devroye and Györfi, 1985). In Bowman (1984) the integrated squared error cross-validation method (i.e., a minimization (2.34) with respect to h) was found to estimate long-tailed distributions by means of variable bandwidth kernel estimators. It was shown in Schuster and Gregory (1981) by a simulation study that this method produces better estimates than the cross-validation for the Cauchy and Student’s t5 distributions. A weighted version of squared error cross-validation for the estimator  f A x h1  h was proposed in Hall (1992). According to this method the empirical

116

HEAVY-TAILED DENSITY ESTIMATION

version of the functional WISE = f˘−i x h2 xdx − 2 f˘−i x h2 fxxdx has to be minimized with respect to h to choose h. Here,   n  x − Xj fˆ−i Xj  h1 1/2 1 1/2 f˘−i x h = fˆ X  h  K nh j=1j=i −i j 1 h

 · 1 x − Xj ≤ Ah  ∀A > 0

(3.22)

(3.23)

and x is a bounded, nonnegative function (a weight). The estimate f˘−i x h is the estimate  f A x h1  h that is calculated over the sample with one excluded observation. For this method the optimal order n−1/9 of h providing the minimum of the MSE was not proved and, hence, the fastest MSE of rate n−8/9 was not obtained. FromTheorem 3.1 of Hall (1992, p. 772) it follows that (in Hall’s notation) Iˆ → I (I =  f A f is a weighted expectation of  f A , Iˆ is an empirical estimate of I) and  → WISE as n → . hence, WISE Novak (1999) and Naito (2001) give estimates that modify fˆhA x and have the same bias Oh4 . In contrast to the retransformed kernel estimators presented below, the variable bandwidth kernel estimators are not intended for the estimation of the PDF at infinity, at least by compactly supported kernels, because the latter estimators are defined on finite intervals which are approximately the same as the ranges of the samples. Example 8 We consider the estimate fˆhA x with kernel given by some symmetric finite PDF, such as the Epanechnikov kernel. Due to the restriction x ≤ 1 of this kernel, we have x − Xi ≤ h/fˆ Xi 1/2 . Let us use the Gaussian PDF N0 ˆ 2  as a pilot estimate fˆ x, where ˆ 2 is the empirical variance. Then the maximal point x where the estimate may be computed is defined by the inequality √  1/2 2 x ≤ Xn + h 2 ˆ exp Xn /2ˆ 2   that is, it depends on the maximal observation Xn . Let us consider the Pareto distribution with PDF  x−+1  x > 1 fx = 0 x ≤ 1

with  = 3. The variance of this distribution is equal to 3/2. The 1 − 10−5 100% quantile is equal to t1−10−5  = 4641, the 95% quantile is t005 = 2714 417 9. If the whole sample falls into the 95% confidence interval and Xn = t005 , then √ 1/2 2 x ≤ t005 + 3 expt005 /3 ≈ 87 as h = 1. At the same time the endpoint of the distribution is approximately equal to 46.41.

HEAVY-TAILED DENSITY ESTIMATION

117

It seems that the selection of a heavy-tailed symmetric PDF, such as Cauchy, as a kernel could extend the ability of variable kernel estimates with regard to the estimation of the ‘tail’.

3.5

Retransformed nonparametric estimators

For finite and light-tailed distributions a histogram is a good estimate of the corresponding PDF. But if the distribution is heavy-tailed, a histogram provides a misleading estimate in the ‘tail’ domain. The same is true for most of the common nonparametric PDF estimates, such as kernel, projection and spline estimates. In general, they have sharp peaks at ‘outliers’ and do not provide the correct rate of decay at infinity (Silverman, 1986). Variable bandwidth kernel estimates with compactly supported kernels are truncated beyond a finite interval that is determined by the largest observation of the sample. It is obvious that nonparametric PDF estimates with good behavior in the ‘tail’ domain are required. This feature is highly significant if PDFs of many populations are compared. Another problem related to the comparison of PDFs is provided by classification (pattern recognition). If one uses an empirical Bayesian classification algorithm, then the observations will be classified by the comparison of the corresponding PDF estimates of each class (Chapter 5). Since the object can arise in the ‘tail’ domain as well as in the ‘body’, a tail estimator with good properties is of great importance for the classification. To improve the behavior of the PDF estimation at infinity one can apply a transform–retransform scheme or a preliminary transformation of the data and estimate the PDF of a new r.v. obtained by the transformation. We discuss here the estimation of a heavy-tailed PDF fx using a transform–retransform scheme that is an alternative to a variable bandwidth kernel estimation. This means that first the X-space data are transformed via the monotone increasing continuously differentiable ‘one-to-one’ transformation function Tx to obtain Y1      Yn Yi = TXi . The derivative of the inverse function T −1 is assumed to be continuous. The DF of Yj is given by Gy = PYj ≤ y = PTXj  ≤ y = PXj ≤ T −1 y = FT −1 y

(3.24)

and its PDF is g0 y = G′ y = fT −1 yT −1 y′ 

(3.25)

Furthermore, the PDF g0 y of Yi is estimated by some estimator gˆ0 y and after back-transformation we get the PDF estimate of the Xi by the formula fˆ x = gˆ0 TxT ′ x

(3.26)

One may take any nonparametric estimator as gˆ0 x. The PDF g0 x should be convenient for the estimation; for example, it should not go to infinity in the domain of definition. The latter can be ensured by the choice of Tx.

118

HEAVY-TAILED DENSITY ESTIMATION

The background to the transformation idea is given by the need for different amounts of smoothing at different locations of a heavy-tailed PDF. Then retransformed PDF estimates with fixed smoothing parameters work like locationadaptive estimates. Therefore, such estimates may better evaluate the tail of heavytailed PDFs. The selection of Tx is an important problem. By (3.24), a transformation Tx is completely determined by the distribution functions Gx and Fx. One can select any ‘target’ Gx, but Tx and Fx are unknown. In Devroye and Györfi (1985) transformations to a finite interval, T  R+ →

0 1, were proposed. It was proved that both the transformation to an isosceles triangular PDF tri x on [0,1], ⎧! ⎨ Fx  Fx ≤ 05 2! T x = ⎩1 − 1−Fx  Fx > 05 2

for kernel estimates with compact kernels, and the transformation Tx = Fx to a uniform PDF uni x for  1 a histogram, provide the minimal convergence rate in metric space L1  ming E 0 ˆg0 x − gx dx. It is remarkable that the metric of the space L1 is invariant with respect to any continuous transformation of the data (see Devroye and Györfi 1985; see also Section 2.1 above):  1

gn x − gx dx

fn x − fx dx = 0

0

Since such Tx and, therefore, the distribution of Yj = TXj  depend on the unknown DF Fx, it is impossible to obtain coinciding values of g0 x and tri x (or uni x). Hence, Devroye and Györfi (1985) proposed using some parametric family of DFs ! instead of F , which depends on some parameter  and to adapt  to the sample.4 However, the concrete models were not indicated and their influence on the decay rate at infinity of the retransformed estimates was not discussed. Wand et al. (1991) and Yang and Marron (1999) consider the families of fixed transformations T x (independent of Fx) given by  if = 0 x sign T x = ln x if = 0  Here,  is the parameter minimizing the functional R g ′′ y2 dy, and gx is the unknown PDF of the transformed r.v. Y1 = T X1  that requires a preliminary  estimation. Since the function R g ′′ y2 dy shows the curvature of the PDF, such transformations are applied for better estimation of curvy but not necessarily heavytailed densities. 4 An empirical DF cannot be used as an estimate of Fx since its derivative does not exist at a finite number of points. Hence, formula (3.26) cannot be applied.

HEAVY-TAILED DENSITY ESTIMATION

119

Markovitch and Krieger (2000) consider the fixed transformation Tx = 2/  arctan x, which provides good accuracy for some heavy-tailed PDFs. However, without assumptions on the type of the distribution any transformation may lead to a PDF that is difficult to estimate from a sample of moderate size and, hence, one cannot provide an accurate estimation of the ‘tails’. To improve the estimation in the ‘tails’ a transformation Tˆ x  R+ → 0 1 which is adapted to the data (via an estimate ˆ of shape parameter ) is proposed in Maiboroda and Markovich (2004). To construct such an estimate Tˆ x of Tx one has to select a target DF Gx and a fitted DF Fx. This adaptive transformation is considered in Section 4.3.

3.6

Exercises

1. Combined estimator. Generate X n with sample size n = 500 according to some heavy-tailed distribution, for example, Fréchet, Weibull with shape parameter less than 1, or Burr. Calculate the combined estimate f˜ t  N  by formula (3.1). For this purpose, use kernel estimator (2.18) with d = 1 as f N t, and the parametric model (3.3) as f t. Estimate the bandwidth h of the kernel estimator by the 2 method, i.e. as a solution of the discrepancy equation (2.41). Use x nh−1 ni=1 − K x − Xi /h dx as Fˆ x. Estimate the shape parameter  by Hill’s method (1.5). Select the number of largest order statistics k by means of a Hill plot (see Section 1.2.2). Use this value of k to determine the ‘boundary’ order statistic Xn−k . Propose a boundary kernel of the f˜ t  N  on the interval Xn−k−1  Xn−k  to avoid the gap between f N t and f t. 2. Repeat Exercise 1 but use the group estimator l to estimate . For this purpose, apply (1.19) and (1.28). Estimate the parameter m (i.e., the number of observations in each group) by means of a plot (see (1.29), Section 1.2.4). Compare the estimates obtained in both exercises. 3. Barron’s estimator (see Section 3.3). Generate X n with sample size n ∈ 50 100 500 1000 according to some heavy-tailed distribution. Consider the following parametric tail models. (a) Lognormal family with PDF 

ln x − "2 gx = √ exp − 2 2 2 x 1





and DF

  1 ln x − " Gx =   x 

for

1 x −t2 /2 e dt x = √ 2 −

120

HEAVY-TAILED DENSITY ESTIMATION

Use two variants of the parameter estimation: (i) The maximum likelihood estimates are "n =

n 1 ln Xi n i=1

n2 =

and

(ii) The moment estimates are m4 1 "n = ln 2 n 2 2 mn + sn

and

n 1 ln Xi − "n 2  n i=1

n2



 sn2 = ln 1 + 2  mn

where mn and sn are the maximum likelihood estimates of the previous variant. (b) Normal family. The maximum likelihood and moment estimates coincide, that is, "n =

n 1 X n i=1 i

and

n2 =

(c) Weibull family with PDF

and DF

 gx = # x#−1 exp −x# 

n 1 X − "n 2  n i=1 i

 > 0 # > 0

 Gx = 1 − exp − x# 

Estimate parameters  and # by the maximum likelihood method. Select the number of intervals mn in Barron’s estimator (3.14), e.g., mn ∈ 5 10 20 . Construct the intervals An1      Anmn by means of the quantiles G−1 j/mn , 1 ≤ j ≤ mn − 1, of the DF Gx. One can select any other auxiliary DF Gx to obtain these intervals. Calculate j , the number of observations falling in each interval Anj . Calculate Barron’s estimate by formula (3.14). Compare the estimation of a PDF using estimates with different tail models for different sample sizes. Draw conclusions regarding the reliability of the estimates for moderate size samples. Investigate how the accuracy of Barron’s estimate depends on the value mn for different sample sizes. 4. Variable bandwidth kernel estimator. Generate X n according to some heavy-tailed distribution. Calculate the estimate by formula (3.15). For this purpose, calculate the auxiliary estimate fˆh1 x by formula (2.18). Select Epanechnikov’s kernel as Kx for both f A x h1  h. Calculate h1 by formula (2.30). Select the estimators fˆh1 x and  parameter h in (3.15) in the following ways:

HEAVY-TAILED DENSITY ESTIMATION

121

(a) Calculate h by formulas (3.20) and (3.21). To do this, first estimate the PDF fx and the derivative d/dx4 1/fx by a kernel method5 . (b) Use the weighted version of the squared error cross-validation method to estimate h (see formulas (3.22) and (3.23)). The choice of  could be  ˆ −1/2 x − " 1 for $ ˆ 2 ≤ z  x = 0 otherwise ˆ denote the sample mean and variance, respectively, where " ˆ and $ · is the Euclidean distance and z is the upper 1 − -level critical point of the  2 distribution with p degrees of freedom,  ∈ 01 02 .

(c) Calculate h by the 2 method, i.e. as a solution of the discrex  pancy equation (2.41). Use nh−1 ni=1 fˆh1 Xi 1/2 − K x − Xi fˆh1  Xi 1/2 /h dx as Fˆ x.

Compare the estimates by different selection methods for h.

5

To estimate the rth derivative of the PDF one can use the estimator fˆ r = n−1 h−r−1

n  i=1

K r x − Xi  /h 

assuming that the kernel function Kx is smooth enough (Wand and Jones, 1995). For an extended discussion on the accuracy of the estimation of the PDF derivative, see Prakasa Rao (1983).

4

Transformations and heavy-tailed density estimation

In this chapter, we study the heavy-tailed PDF estimates based on the preliminary data transformations. Fixed and adaptive transformations are considered. To improve the behavior of a retransformed kernel estimate at infinity, boundary kernels are studied. To select smoothing parameters of the nonparametric PDF estimators the data-dependent discrepancy methods are investigated. These methods are applied both to nonvariable and variable bandwidth kernel estimators as well as to a projection estimator. The mean squared errors of these estimates are proved to be optimal.

4.1

Problems of data transformations

It is well known that kernel estimates provide a good asymptotic MISE and MSE for sufficiently smooth PDFs. For instance, the variable bandwidth kernel estimates give MISE ∼ n−8/9 and MSE ∼ n−8/9 even without a preliminary transformation of the data if the bandwidth h is taken proportional to n−1/9 . However, this does not imply that the estimation in the tail domain will be good enough. Relatively large values of the PDF in the body generate the main contribution in the MSE (MISE), unlike small values in the tail. Hence, the MSE and MISE as well as popular

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

124

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

measures of the metric spaces C, L1 and L2 are not sensitive to the accuracy of the estimation at the tail. For samples of moderate size a preliminary data transformation may improve the estimation of heavy-tailed PDFs at infinity (Section 4.3) or curvy PDFs (Section 3.5). The MISE of the retransformed PDF estimates is determined by the MSE of the PDF estimate of the new r.v. which is constructed by the transformation (Section 4.6). Fixed transformations that do not require any knowledge of the distribution type, for example ln x or 2/ arctan x, are more attractive for practical applications (see Section 4.2). Nevertheless, they may result in discontinuous PDFs of transformed r.v.s, which are difficult to estimate. In general, the tail of the PDF cannot be estimated accurately by pure nonparametric methods since without imposing assumptions on the tail behavior the shape of the PDF of the transformed r.v. cannot be predicted. Considering kernel estimates, the rate of decay of retransformed estimates in the distribution tail may be close to that of the true PDF for certain boundary kernels and appropriate bandwidths (Section 4.5). To improve the accuracy of retransformed estimates, the selection of the smoothing parameter (e.g., the kernel estimate bandwidth or the polygram bin width) constitutes the most important problem. It is better to estimate this parameter for the PDF of the transformed r.v. if the latter is compactly supported and sufficiently smooth. Data-driven selection methods like the cross-validation and the D and 2 discrepancy methods are universal in the sense that they are applicable to any nonparametric PDF estimator. It is proved that the D method may provide the optimal rates of the MSE of kernel estimates with variable and nonvariable bandwidths (Sections 4.7 and 4.8), while the 2 method provides an optimal convergence rate of some projection estimate in the metric of the space L2 (Section 4.9). The proofs of all stated theorems are presented in Appendix B.

4.2

Estimates based on a fixed transformation

Here, the fixed transformation arctan x is considered. The estimates obtained by means of this transformation were investigated in a Monte Carlo study (Markovitch and Krieger, 2000). The transformation Tx =

2 arctan x 

T ′ x =

2 1 + x2 

(4.1)

does not depend on the sample X n and satisfies the conditions on transformations assumed in Section 3.5. Tx generates an r.v. Y = TX with a bounded PDF1 1

gx is calculated by formula (3.25).

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

125

6

f(x)

4

2

0

0

0.4

0.2

x

0.6

0.8

1

Figure 4.1 PDFs of transformed r.v.s Y generated by transformation (4.1) for different fx: standard exponential (solid line), Cauchy (solid horizontal line), Weibull with shape parameter 0.5 (dotted line), gamma with shape parameter 2 (dot-dashed line), lognormal with parameters (1, 1) (solid line with + marks). Based on Figure 1 in Markovich and Krieger (2000).

gx for many heavy-tailed PDFs fx (apart from the Weibull distribution; see Figure 4.1). The estimate gn x may not obey the conditions of a PDF on [0, 1] since part of the distribution may be located outside [0, 1]. However, one may normalize it, that is, use the estimate gˆ n x =  1 0

gn x gn xdx

instead of gn x. The risk in the metric space L1 will decrease after such normalization. This implies, for the estimate fˆn x = gˆ n TxT ′ x

(4.2)

that 

 0

fˆn x − fxdx =



1 0

ˆgn x − gxdx ≤



1 0

gn x − gxdx

(Devroye and Györfi, 1985). The following algorithm to estimate a heavy-tailed PDF is considered: • Construct the nonparametric estimate gn , located on [0,1], from the transformed sample Y n = Y1      Yn , Yi = TXi , i = 1     n, and normalize if necessary.

126

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

• Calculate an estimate of the smoothing parameter of gn . • To obtain the estimate of the PDF fx, apply an inverse transformation (4.2). For the purpose of the analysis the polygram (2.29) and kernel estimators with the Gaussian kernel and Epanechnikov’s kernel (2.21) are used. For transformation (4.1) these kernel estimates of the transformed r.v. Y1 are determined by    n  1 x − Yi 2 1 1 ghn x = √ exp − (4.3) 2 h nh 2 i=1

and

2 ghn x

respectively, where

   n x − Yi 2 3  = 1−

h + Yi − x 4nh i=1 h

2 Yi = arctanXi  

t =

(4.4)



1 t ≥ 0 0 t < 0

By (4.2) we obtain, after normalization, the final PDF estimates constructed from the transformed sample Y n : ⎛  2 ⎞ √ 2 n  arctanx − Y 1 2 i 1  ⎠ fˆhn x = exp ⎝− (4.5) 3 1 2 h h1 + x2  i=1 nh 2 I 01 and

2 fˆhn x =

Here

n 

⎛

3 ⎝1 − 2 2nhI 01 h1 + x2  i=1   2 · h + Yi − arctanx   1 I 01 h =



2 

arctanx − Yi h

   n   1 − Yi Y 1  − − i n i=1 h h

2 ⎞ ⎠

(4.6)

 2 √ x 1 is the integral of ghn x on 0 1 , x = 1/ 2 − exp − u2 du is the Gaussian DF and n 1 − 3h1 2 1 for h + Yi > 1 3  2  i − 1   + 3Y2i Y I 01 h = Yi 2 4nh i=1 3 h + Yi 1 − 3h2  for h + Yi ≤ 1 2

is the integral of ghn x on [0,1].

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

127

Let gLn x be a polygram constructed on Y n by formula (2.29). After the inverse transformation (4.2) we get (since no normalization is necessary)   2 2 fLn x = g arctan x  (4.7) 1 + x2  Ln 

Let us now discuss the selection of the bandwidth h determining the accuracy of 1 the kernel estimates and a polygram. The parameters h and L of estimates ghn x, 2 n ghn x and gLn x may be calculated from a sample Y using the cross-validation method (2.31) (or (2.35) for the Gaussian kernel) or the 2 and D discrepancy methods (see (2.38)–(2.40)). The idea of the 2 method is to obtain h (or L) from  n   1 i − 05 2 2 h = 005 +  ˆ n h = G Yi  − n 12n i=1 or, in the case of the D method, from √ √ ˆ n = n maxD ˆ n+  D ˆ n−  = 05 nD

where   √ + √ i h ˆ − G Yi   nDn = n max 1≤i≤n n

√

ˆ n− = nD

√

 i−1  n max G Yi  − 1≤i≤n n 

h

and Y1 ≤ Y2 ≤    Yn are the order statistics of the transformed observations. For the normalized kernel estimate (4.3) we get     x n    1 x − Yi 1 Y 1  Gh1 x = 1 ghn tdt = 1 − − i  h h I 01 h 0 nI 01 h i=1 while for the normalized kernel estimate (4.4) we get  x 1 2 Gh2 x = 2 g tdt I 01 h 0 hn   n  3 x − 3h1 2 x − Yi 3 + Yi3    = 2 4nhI 01 h i=1 x − 3h1 2 h3 + Yi3 

h + Yi ≥ x h + Yi < x

For the polygram one can calculate

GL x =



x 0

gLn t dt

instead of Gh Yi  in the formulas above. In Markovitch and Krieger (2000) the polygram and kernel estimates (4.5) and (4.6) with noncompact and compact kernel functions were compared for long-tailed distributions by a simulation study. Moreover, the 2 and D methods were compared with the cross-validation method on p. 77. Regarding the comparison, samples of a gamma distribution with parameter s = 2, a lognormal distribution with  = 1  = 1 and a Weibull distribution

128

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

with s = 05 were generated. The gamma distribution is related to lighttailed distributions, but the lognormal and Weibull PDF are heavy-tailed. As characteristics of the estimates the loss functions in metric spaces L1 , L2 and C were used. The simulation study showed that the heavy-tailed Weibull PDF is difficult to estimate using such retransformed kernel estimates. For this PDF there is no uniform convergence for any of the estimates and smoothing methods considered. At the same time, the polygram demonstrates better accuracy than kernel estimates in L1 and L2 . For the gamma and lognormal PDFs the polygram and the kernel estimate with the Gaussian kernel are preferable. They provide convergence in all metrics considered, whereas the kernel estimate with Epanechnikov’s kernel does not converge in C for the lognormal PDF as the sample size increases. It follows from the simulation study that a polygram and kernel estimate (4.5) are preferable for the application to real data if the true PDF is not available. If one knows that the PDF is heavy-tailed, then a polygram may be recommended.

4.3

Estimates based on an adaptive transformation

Some important questions about the transform–retransform scheme (see Section 3.5) may arise: firstly, what family of distributions is a reasonable approximation of the true DF; secondly, what is a better target DF of the transformation considering the stability of the retransformed estimates to minor perturbations in the transformation; and thirdly, which nonparametric estimate best maintains the rate of tail decay of the true PDF. Here we discuss all these questions. The adaptive transformation described in the following is derived from a specific assumption regarding the parametric model  of the DF Fx; see Maiboroda and Markovich (2004). The system of heavy-tailed distributions is taken as  , where the EVI  is estimated using Hill’s estimator.

4.3.1

Estimation algorithm

The algorithm proceeds as follows: • Estimate the EVI of Xj from the sample X n (see Section 1.2), for example, using Hill’s estimator ˆ n =   H n k.

• Construct the transformation T = Tˆ n in the following way: if  has the fitted DF ˆ n then Tˆ n  has the target DF , for example, a uniform or triangular one. (Here ˆ n is considered as a fixed value.)

• Construct the transformed sample Yj = Tˆ n Xj  j = 1     n.

• Estimate the PDF of Y1      Yn by some estimator gˆ n x. • Estimate the PDF of Xj by (3.26).

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

4.3.2

129

Analysis of the algorithm

Given a realization of the algorithm, we must choose a family of DFs, the target distribution, and the estimate gˆ n . The family of fitted DFs must be chosen in such a way that the transformation T and its derivative can be easily evaluated. If ˆ is the fitted DF and  is the target DF, then by (3.24), Tx = −1 ˆ x and T −1 x = ˆ−1 x hold. We assume that the GPD ˆ −1/ˆ  1 − 1 + x if x ≥ 0 ˆ x = (4.8) 0 if x < 0 with  > 0 is chosen as a fitted DF. Evidently, the fitted PDF is determined by ˆ ˆ −1/+1 ˆ x = ˆ′ x = 1 + x 

(4.9)

This choice of distribution is widespread and motivated by the theorem of Pickands (1975); see Section   1.1. This theorem states that, for a certain class of distributions Fx ∈ MDA H ,  ∈ R, of the r.v. X and for a sufficiently high threshold u of the r.v. X, the conditional distribution of the overshoot Y = X − u, provided that X exceeds u, converges to the GPD. We consider the uniform PDF uni x = 1x ∈ 0 1 and the positive triangular PDF +tri x = 21 − x1x ∈ 0 1 as our target PDFs. The corresponding DFs are therefore uni x = x1x ∈ 0 1 + 1x > 1 

+tri x = 2x − x2 1x ∈ 0 1 + 1x > 1 

We wish to clarify two questions:

• Which transformation ensures a more stable estimation algorithm given deviations of an EVI estimate? • What tail behavior is ensured by the inverse transformation? Let the target be  = uni   = uni . Then

ˆ −1/ˆ  Tˆ x = −1 ˆ x = 1 − 1 + x

ˆ Tˆ−1 x = 1 − x−ˆ − 1/

ˆ ˆ −1/+1 Tˆ′ x = 1 + x 

ˆ Tˆ−1 x′ = 1 − x−−1 

We suppose that the true PDF of X is given by

fx = ℓx x

(4.10)

where ℓx is a bounded function with limx→ ℓx = ℓ <  ( is the EVI of this PDF). Then by (3.25),   −1/+1  1 − x−ˆ − 1 1 − x−ˆ − 1 ˆ −−1 gx = 1 +  1 − x ·ℓ  x ∈ 0 1 ˆ ˆ

130

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

follows as the PDF of Y = Tˆ X. This PDF will be estimated by gˆ n x for some ˆ Hence, its behavior on its support x ∈ 0 1 must be convenient for the value of . estimation. If x is close to zero, then gx is bounded. Let us consider gx as x ↑ 1. Note that in this case ℓTˆ−1 x → ℓ <  since Tˆ−1 x → . Then for x ↑ 1 it follows that  −1/+1 1 − x−ˆ − 1 ˆ gx ≃ ℓ 1 +  1 − x−−1  ˆ For  = ˆ we have gx ≃ ℓ , that is, gx behaves as a uniform PDF in the neighborhood of x = 1 (up to an insignificant multiplier ℓ ). ˆ For x ↑ 1 and any constant C < , we have 1 − x−ˆ + C ∼ Let  = . −ˆ 1 − x . Hence, ˆ ˆ −1/+1 1 − x/−1 gx ≃ ℓ / 

ˆ − 1 > 0, then the PDF Therefore, if ˆ overestimates the true EVI , that is, / gx behaves nicely in the neighborhood of x = 1 (provided it is not like the uniform distribution). But if we underestimate  ˆ <  then gx →  as x ↑ 1. In such situations PDF estimates perform very poorly, since they are designed for the estimation of finite values. We now suppose that the target PDF is a triangular one. Then √ −1 x = +tri −1 x = 1 − 1 − x ˆ ˆ −1/2  Tˆ x = 1 − 1 + x ˆ ˆ −1/2−1 Tˆ′ x = 051 + x 

ˆ Tˆ−1 x = 1 − x−2ˆ − 1/

(4.11)

ˆ Tˆ−1 x′ = 21 − x−2−1 

If the PDF of X is of the form (4.10), then ˆ ˆ −1/+1 1 − x2/−1 gx ≃ ℓ / 

as x ↑ 1

arises as the PDF of Y = Tˆ X. For ˆ =  the PDF gx → 0 as x ↑ 1 and its rate of decay is the same as for the triangular PDF, ∼ C1 − x. If ˆ =  and ˆ > /2 then gx tends to zero as x ↑ 1. The rate of this decay is not asymptotically equal to C1 − x, but PDF estimates will normally work in this case. These problems can only arise if ˆ < /2. If  is estimated by a consistent estimator (e.g., by Hill’s) such rough underestimation will be a rare event. In fact, the probability of getting a value ˆ which is far from  tends to zero as the sample size n ↑ . We suppose now that we have estimated gx by gˆ n x. What is the value of the EVI if the PDF estimate fˆn x is defined by (3.26)? Let gˆ n x be a histogram (or a polygram) estimate. Since for both transformations Tˆ x → 1 as x → , we have fˆn x ≃ gˆ n 1Tˆ′ x as x →  by (3.26). Hence, the accuracy of the estimation in the tail depends on the behavior of the estimate gˆ n x in the neighborhood of x = 1. We suppose that gˆ n 1 > 0. This property holds almost surely for large n if R+ is the support of X. Note that gˆ n 1 is the height of the rightmost bar of the histogram.

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

131

ˆ ˆ −1/+1 If the target PDF is uniform, this means that fˆn x ≃ gˆ n 11 + x , that is, the estimate has the EVI which we have chosen for the fitted PDF. In our ˆ algorithm it is given by . The same is true for a kernel estimate when gˆ n 1 is replaced by gˆ n 1 h. The kernel estimate with a compactly supported kernel (e.g., Epanechnikov’s kernel (2.21)) gˆ n x depends on the smoothing parameter h and gˆ n 1 h ∼ Oh−1  for sufficiently large h > 1 − Tˆ Xn , whereas gˆ n 1 h = 0 holds for other h. The situation can be explained by Figure 4.2. Here, we observe boundary effects of the kernel estimate applied to a PDF with a compact support due to the truncation of the kernel near the boundaries. If the target PDF is triangular, we have ˆ ˆ −1/2+1 fˆn x ≃ 05ˆgn 11 + x 

In this case, the EVI is twice as large as needed. To remove this effect, we can use a smoothed histogram (or polygram). For such gˆ n x we have gˆ n x = Cn 1 − x for x ≃ 1, that is gˆ n x ≃ +tri x. Here Cn is the slope of the line which connects the center of the top of the rightmost histogram bar with the point (1,0). For the triangular target PDF we then get

8

f(x)

6

4

2

0 0.8

1.2

1

1.4

x

Figure 4.2 Kernel estimate with Epanechnikov’s kernel in the neighborhood of 1: h1 < 1 − Tˆ Xn  (solid line), h2 = 1 − Tˆ Xn  (dotted line), h3 > 1 − Tˆ Xn  (dashed line), Tˆ Xn  = 08. Reprinted from Computational Statistics, 19(4), pp. 569–592, Estimation of heavy-tailed probability density function with application to Web data, Maiboroda RE and Markovich NM, Figure 1, © 2004 Physica-Verlag, A Springer Company. With kind permission of Springer Science and Business Media.

132

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION ˆ ˆ ˆ −1/2 ˆ −1/2+1 fˆn x ≃ 05Cn 1 − 1 − 1 + x 1 + x

≃

Cn ˆ ˆ −1/+1 1 + x  2

so that the EVI of the estimate coincides with its estimate of the fitted PDF. For a kernel estimate this rate in the tail may be given by a correct selection of a kernel near the boundary and a smoothing parameter h. One can take the triangular kernel Kx = 1 − x1x ≤ 1 as a boundary kernel. Then, by (2.18), gˆ n x ≃ 1 − x − Tˆ Xn /h/h holds for boundary points x ∈ Tˆ Xn  1 . For h = 1 − Tˆ Xn  it follows that gˆ n x ≃ 1 − x/h2 . Therefore, we get the same tail of fˆn x as for a smoothed polygram. The estimation of the body and the tail of the PDF of the Fréchet(0.3) distribution (1.32) by the retransformed kernel estimate with Epanechnikov’s kernel and the retransformed polygram is shown in Figure 4.3. The sample size is n = 50. Here, h = n−1/5 = 0457 > h1 = 101 − Tˆ Xn  = 0112. The maximal observation Xn ˆ in the sample is equal to 9.235. Hill’s estimate n k of the EVI is equal to 0.278, where k = 11 is obtained by the bootstrap method (Caers and Van Dyck, 1999) with resample size B = 50. The parameter L of the polygram is equal to 15. The tail domain is shown on a logarithmic scale on both axes. The value h1 provides a better estimation of the tail of the PDF than h due to the better estimation of the PDF gx of the transformed r.v. at the boundary. Experience tells us that the triangular and Epanechnikov’s kernels provide similar results for the same h. If a value of h smaller than h1 is used, it leads to truncation of the kernel (see Figure 4.2). Hence, the estimate is equal to 0 in the tail.

0.6

f(x)

f(x)

0.4

0.2

0

0

5 x

10

0.1 0.01 . 1 10–3 1 .10–4 1 .10–5 1 .10–6 1 .10–7 1 .10–8 1 .10–9 1 .10–10 1 .10–11 1 .10–12 1 .10–13 1 .10–14 1 .10–15 1 .10–16 10

1.103

100

1.104

x

Figure 4.3 ‘Body’ (left) and ‘tail’ (right) estimation of a Fréchet PDF (solid line) by retransformed estimates: kernel estimate with h (dotted line), kernel estimate with h1 (solid line with + marks), polygram (dashed line). The polygram nearly coincides with the PDF in the tail domain. The kernel estimate with h1 is best in the ‘body’.

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

133

Generally speaking, a boundary kernel should coincide with a target PDF, which is nearly the same as gx for ˆ ≈ . Let us explain why the polygram is preferable to the histogram in this case. As a matter of fact, a PDF g with gx → 0 as x → 1 is estimated. If the lengths of the bins are equal, very few observations fall into the rightmost bin. Hence, the histogram estimate is not stable in the tail. The polygram dynamically adapts the bin widths to the data and works better. Of course, the algorithm considered can perform in inadequately if the underlying PDF has light tails. Therefore, it is a good idea to test this hypothesis before the PDF is estimated. A comprehensive list of references on such tests can be found in Jureˇcková and Picek (2001) and Dietrich et al. (2002). It is useful to provide a preliminary data analysis as is done in Section 1.3. In summary, we conclude that among the alternatives considered the best combination of parameters of the algorithms is determined by +tri as target PDF and a smoothed polygram (or a kernel estimate with compactly supported kernel) as PDF estimate on [0,1]. The next question concerns the outcome if one applies the transformation (4.11) but the true PDF does not belong to a Pareto class. Typical distributions with heavy tails are distributions with regularly varying tails (like (4.10)), lognormal-type tails and Weibull-like tails (Mikosch, 1999). Applying the transformation (4.11) to the lognormal and Weibull PDFs, one can conclude by (3.25) that the corresponding PDF gx of the transformed r.v. is continuous in the neighborhood of x = 1. However, without the assumption on the class of the tail, an accurate estimation of the tail by means of a nonparametric method is impossible, because in this case one cannot select a suitable boundary kernel.

4.3.3

Further remarks

Our first aim here has been to present a nonparametric method which provides a good estimation of the ‘tails’ of heavy-tailed PDFs. The transformation to a triangular PDF and a smoothed polygram (or a kernel estimate) is established as the best combination of an estimation. It allows us to get a stable estimate of a PDF with regard to small perturbations of the transformation due to a rough EVI estimation, and to retain the tail decay of the true PDF after the inverse transformation. Retransformed nonparametric estimates have the following advantages: 1. Estimates with a fixed smoothing parameter h work like estimates with a variable h. 2. The data transformation allows us to apply nonparametric (histogram, polygram, projection) estimates which are only suitable for PDFs with finite support to improve the accuracy of the estimates for heavytailed PDFs.

134

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

3. The transformation approach is the most suitable one for heavy-tailed PDFs or PDFs with sharp features, such as high skewness (Wand et al., 1991; Wand and Jones, 1995; Yang and Marron, 1999). It makes no sense to apply the transformation approach to ‘simple’ PDFs. Such PDFs may be better estimated by standard nonparametric methods. The accuracy of a retransformed estimate is defined by the accuracy of the PDF estimator of a transformed r.v.; in particular, the mean integrated absolute error (MIAE) is invariant to the transformation (see property (2.2)), so MIAEx = MIAETx (Devroye and Györfi, 1985). The latter may be worse than the accuracy of an untransformed PDF estimate. For example, suppose that a Gaussian r.v. is transformed into a triangular distributed r.v. It was shown in Markovich (1989) that a kernel estimator with the Gaussian kernel estimates the normal PDF better than a triangular one. It is clear that one can also give examples of the preference of the transformed approach. An alternative approach to estimating heavy-tailed PDFs is determined by variable bandwidth kernel methods (Abramson, 1982; Devroye and Györfi, 1985). However, the latter estimators are not intended for accurate tail estimation. In order to recognize the heaviness of the tail one may use one of the tests specified in Jureˇcková and Picek (2001). Simplicity of calculations is very important in practice. In this respect the use of fixed transformations (like that in Section 4.2 or the classical Tx = ln x), leading to bounded, convenient PDFs of a transformed r.v. that require no assumptions on the distribution, may be sometimes preferable. However, even if information about the behavior of the distribution at the tail is available, fixed transformations cannot provide PDFs of the transformed r.v.s with predictable features. The latter may lead to bad estimation of the PDF. In this respect, adaptive transformations are more flexible. Here, a transformation that is adapted to the data under the assumption that the true distribution belongs to a Pareto class is considered. In Section 4.3.2, PDF estimation in the class of distributions with regularly varying tails (4.10) is investigated. Due to the choice of ℓx this class is rather wide and includes the Pareto distribution as well. We leave the consideration of heavy-tailed PDFs with other types of tails as an open problem. The Pareto parameter (or an EVI) reflects the shape of the tail and is estimated by sparse data. It may also be estimated by Hill’s estimator. The fitting capabilities of a transformation family strongly depend on the accuracy of Hill’s estimator. It is known that the latter is accurate only for sufficiently large sample sizes. Therefore, more accurate EVI estimates could be preferable. To provide the correct tail decay at infinity the smoothed polygram is used. The selection of a boundary kernel and a smoothing parameter for it is proposed for a kernel estimate. A smoothed polygram may be preferable, especially for limited sample sizes and for tail estimation. For sufficiently large samples, kernels give a better estimate for the ‘body’ of a PDF. The quality

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

135

of the kernel estimates may be improved by further boundary corrections to improve the tail estimation, a better EVI estimation, and a better smoothing procedure.

4.4

Estimating the accuracy of retransformed estimates

We consider the MISE on the interval , ˆ  = E MISEh  =E =E





fˆ x − fx2 dx



ˆgh Tˆ x − gTˆ x2 Tˆ′ xdTˆ x





∗

(4.12)

ˆgh y − gy2 Tˆ′ Tˆ−1 ydy

as a measure of the quality of the estimate fˆ x. Here ∗ = Tˆ  and gx is the PDF, gx = fTˆ−1 xTˆ−1 x′ 

(4.13)

which is actually estimated instead of g0 x = fT−1 xT−1 x′ (since ˆ = ), and gˆ h x is some estimate of gx with the smoothing parameter h. For fixed transformations and nonrandom intervals ∗ the MISE has a simpler form than (4.12): MISEh  =



∗

T ′ T −1 yEˆgh y − gy2 dy

(4.14)

If 0 < T ′ T −1 x ≤ c holds on ∗ for the transformation T (not necessarily fixed), then we have  MISEh  ≤ c Eˆgh y − gy2 dy (4.15) ∗

for a nonrandom ∗ . This means that the order of the MISE of the retransformed estimates at  is at least not worse than the order of the MSE of gˆ h y. For example, a kernel estimator or a polygram can be used as gˆ h x. Example 9 Note that for the adaptive transformation (4.11), the function Tˆ′ Tˆ−1 x = 05 1 − x1+2ˆ is bounded on [0,1].

136

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

4.5

Boundary kernels

If the tail shape is known, then, irrespective of the transformation, the order of decay of transformed estimates on tails may be close to that of the true PDF fx for a proper choice of kernels and smoothing parameter h near the bounds of variation of the transformed r.v. Y1 = TX1 , (Markovich, 2005c). Let us assume that the PDF of the r.v. X1 belongs to class2 ℓx1 + x−1/+1  x ≥ c > 0 (4.16) fx = 0 x < c where ℓx is a slowly varying function (see Definition 11). We consider transformation (4.11) first and obtain the PDF estimate for the r.v. X1 , ˆ ˆ −1/2−1 fˆ x = gˆ h Tˆ xTˆ′ x = 05ˆgh Tˆ x1 + x 

(4.17)

ˆ ˆ −1/−1 We need to find fˆ x ∼ 1 + x , which is close to (4.16) for a sufficiently ˆ Taking (4.17) into account, the accuracy of the tail of the accurate estimate . estimate fˆ x depends on the behavior of the estimate gˆ h x near x = 1 since Tˆ x → 1 as x → . It is known that the kernel estimates (2.18), when they are applicable to PDFs, concentrated on bounded intervals, exhibit boundary effects due to the truncation of kernels at the support boundary (this is discussed in detail in Section 4.3 and in Figure 4.2). This effect is suppressed with boundary kernels. They are useful in improving the estimation of PDFs that take zero values or suffer from discontinuity (a bounded jump) at the boundary. To overcome   this effect, Scott 2 2 (1992) used modified bi-weight kernels Kb x = 15 1 − x . Schuster (1985) 16 + studied the mirror imaging of the sample to negative values. These methods can be successfully applied only if the location of the discontinuity or zero value of the PDF is known. In general, such information can be obtained from a histogram. A boundary kernel is constructed under certain constraints on the kernel. For example, the kernel and its derivative vanish at the boundary if the PDF vanishes. Since such a boundary kernel does not depend on the PDF, there is no guarantee that the estimation would be good. Following Simonoff (1996), the bias of the estimate at the boundary can be reduced by using a linear combination B1 or B2 of two kernels Kx and LxKx is the usual kernel used in PDF estimation at interior points of an interval and is defined on [0,1], whereas Lx is a kernel differing from Kx, but related to it in some way):

B1 x =

2

c1 pKx − a1 pLx  c1 pa0 p − a1 pc0 p

The value x is bounded away from zero to avoid problems, particularly if ℓx = ln x.

(4.18)

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

where al p = or



1 1−p

ul Kudu

B2 x =

cl p =



1 1−p

ul Ludu

a2 p − a1 px Kx  a0 pa2 p − a21 p

137

0 < p < 1

(4.19)

if Lx = xKx. This is only possible if the second derivative of the estimated PDF is continuous. The bias of a kernel estimate with kernel B1 or B2 in the boundary domain is of order Oh2  and the variance is of order Onh−1  (the same as at the interior points of the interval). Now we must find such boundary kernels that allow us to get for retransformed estimates the same decay rate at infinity as for the true PDF. This is not possible unless the shape of the distribution tail is known. Since the approaches described above do not use such information, they are not always useful for this purpose. In particular, the order of decay of the tail of the retransformed estimate (2.18) with kernel B2 x depends on the kernel Kx for class (4.16) and transformation (4.11). Expression (4.17) shows that the kernel must be chosen such that ˆ ˆ −1/2 gˆ h Tˆ x ∼ 1 + x 

as x → 

(4.20)

to obtain an order of decay at infinity close to that of (4.16). This can be done in two ways (Markovich, 2005c): (i) the kernel at boundary points Yn  1 in the kernel estimate gˆ h x must be chosen equal to the ‘target’ PDF gx, or (ii) for an arbitrary kernel the parameter h must be determined from the equation   Tx − Yn 1 K T ′ x = fˆ x (4.21) h h where Yn is the maximal order statistic corresponding to the transformed sample.

Let us consider the first item. For a kernel estimate we obtain at the boundary points y ∈ Yn  1 ,   y − Yn 1  gˆ h y ≃ K h h

that is, gˆ h y is approximated at boundary points by the last term of the sum (2.18). From (4.11) we find for a triangular kernel Kx = 21 − x1x ∈ 0 1 coinciding with the ‘target’ PDF and for the Epanechnikov kernel, that   Tˆ x − Yn ˆ ˆ −1/2 K ∼ 1 + x  as x →  h

138

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

if h = 1 − Yn . Use of B2 x in gˆ h y gives (4.20) for h = 1 − Yn , that is,   Tˆ x − Yn 1 gˆ h Tˆ x ≃ B2 h h     Tˆ x−Yn Tˆ x−Yn a p − a p K 1 h h 1 2 ˆ ˆ −1/2 = ∼ 1 + x  2 h a0 pa2 p − a1 p Then by (4.17) we obtain

ˆ ˆ −1/+1 fˆ x ≃ 1 + x 

Hence, the tail of the estimate fˆ x coincides with the tail of (4.16) both for triangular and Epanechnikov’s kernels if ˆ is close to .3 But for the biweight kernel Kb x,   Tˆ x − Yn ˆ −1/ˆ  K as x →  ∼ 1 + x h and (4.20) is not satisfied. According to the second item, h can be found at the boundary by the equality   Tˆ x − Yn 1 ˆ ˆ ˆ −1/2−1 ˆ −1/+1 B 1 + x = 1 + x  h 2 h

Let us examine the transformation Tx = 2/ arctan x. The PDF of the transformed r.v. is defined by (3.25) as   x   1 + tan2 /2x gx = ℓ tan  (4.22) 2 2 1 +  tan/2x1/+1

gx is continuous in [0,1] if  ≤ 1. If  > 1, then gx →  as x → 1 and it is not easy to estimate gx near x = 1. Moreover, g ′′ x is continuous on [0,1] for some ℓx if  ≤ 1/3.4 In the latter case, the estimate with kernel B1 or B2 could be applied to gx at boundary points to reduce the bias. Otherwise, one can use boundary kernels similar to those recommended in Scott (1992). In any case, since Tx does not depend on , the estimate   2 fˆ x = 2ˆgh arctan x /1 + x2  (4.23) 

does not depend on  if Kx or h is chosen independently of . Then the necessary order of the decay for class (4.16) is not ensured. 3

The parameter  can be found by some other method from Section 1.2.   ′′   The derivative ℓ tan x is not continuous for ℓ∗ x = exp ln1 + x1/2 cosln1 + x1/2  , 2 unlike such ℓx where limx→ ℓx = ℓ  0 < ℓ < , (e.g., positive constants or functions converging to positive constants) and ℓx = ln x. The function ℓ∗ x is an example of a slowly varying function that oscillates at infinity, that is, limx→ inf ℓx = 0 and limx→ sup ℓx =  (Mikosch, 1999).

4

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

139

Choosing the boundary kernel Ky =

 1 + tan2 y/2  ˆ 2 1 + ˆ tany/21/+1

equal to gx, we obtain for any  > 0,

y−Y

gˆ h y ≃

1 + tan2  2 h n    ˆ 2h 1 + ˆ tan  y−Yn 1/+1 2

h

Let h = 1 − Yn /2 arctan x for some x > tan/2Yn . Then (4.23) tends to the tail of (4.16) as x → . Here the window width h depends on x. The last kernel is somewhat artificial since it is similar to (4.22). We can select a -dependent bandwidth h for the usual kernel Kx to ensure a proper order of decay at infinity. For example, if  ≤ 1/3, then at the boundary h can be found from the equality   2/ arctanx − Yn 2 ˆ ˆf x ≃ 1 B2 ˆ −1/+1 = 1 + x  h h 1 + x2  The algorithm for boundary kernel selection is as follows:

1. Select an appropriate transformation function Tx  R+ → 0 1 .

2. Determine the class of the distribution of r.v. X.5

3. Determine the PDF gx of the retransformed r.v. Y = TX by formula (3.25). 4. Determine the kernel Kx at boundary points Yn  1 coinciding with gx. An alternative is to select h for some Kx from (4.21). 5. Use kernels B1 or B2 if g ′′ x is continuous.

4.6

Accuracy of a nonvariable bandwidth kernel estimator

We consider a nonvariable bandwidth kernel estimator fˆh x = nh−1

n  i=1

Kx − Xi /h

(4.24)

where the kernel K  R → R is a real function, h is its bandwidth. Let Kx be symmetric and vanish outside a compact set. For simplicity, we assume that Kx

5 The class of the tail can be determined by some parametric test (Jureˇcková and Picek, 2001). Very roughly, one can find it by the investigation of the sign of the tail index or EVI.

140

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

is defined on −1 1 . Suppose, that Kx is of second order (see Definition 14),6 that is,   Kxdx = 1 xKxdx = 0  x2 Kxdx = K1 = 0

The derivative of the PDF fx is assumed to be continuous and its second derivative is bounded. We write  K1f x = u2 Kuf ′′ x + hu 1u ≤ 1 du 0 < < 1

To get the bias of fˆh x we use the substitution y − x/h = u, the symmetry of Kx (Kx = K−x) and apply the Taylor expansion to f x + hu. It follows that biasfˆh x = E fˆh x − fx (4.25)     1 x−y = fy1x − y ≤ h dy − fx Ku1u ≤ 1 du K h h  = f x + hu − fx Ku1u ≤ 1 du

Let

   K f x hu2 ′′ ′ f x + hu Ku1u ≤ 1 du = h2 1  huf x + = 2 2

K∗ = K2f x =





K 2 u1u ≤ 1 du

K1∗ =



uK 2 u1u ≤ 1 du

u2 K 2 uf ′′ x + hu 1u ≤ 1 du

0 < < 1

Applying the Taylor expansion to f x + hu, we then get the variance of fˆh x:    2 var fˆh x = E fˆh2 x − E fˆh x (4.26)  x−y = n−1 h−2 K 2 fy1x − y ≤ h dy h  2 − n−1 E fˆh x − fx + fx   f  −1 2 −1 2 K1 x = nh K ufx + hu1u ≤ 1 du − n fx + h 2 6

The simplest example of such kernels is given by a symmetric kernel with compact support.

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

= nh

−1



fxK

∗

K f x + hK1∗ f ′ x + h2 2 2

 = nh−1 fxK ∗ + O n−1  



−n

−1



fx + h

141

f 2 K1 x

2



Hence, the MSE for a nonvariable bandwidth kernel estimate (4.24) with secondorder kernel obeys MSEfˆh  = Efˆh x − fx2  2   = h4 K1f x /4 + nh−1 fxK ∗ + O n−1 

(4.27)

Obviously, MSE ∼ n−4/5 if the bandwidth is h = hn = Dn−1/5 , where D depends on the unknown PDF fx, f ′′ x, and the kernel Kx.

4.7

The D method for a nonvariable bandwidth kernel estimator

For moderate sample sizes, a data-dependent choice of the smoothing parameter of the PDF estimate is a more practical tool than one derived from theory such as h = hn = Dn−1/5 , where D is a positive constant. A well-known data-dependent method is given by cross-validation. However, this method has slow convergence rates and high sampling variability (see Park and Marron, 1990). An alternative data-dependent smoothing tool is determined by the discrepancy method (see Section 2.2.4). Here, we shall find the rate of h for the nonvariable bandwidth kernel estimate fˆh x (4.24), when h is defined by the following version of the discrepancy equation. Let the bandwidth h be selected from the discrepancy equation sup Fn x − Fh x = n− 

as 0 <  < 1/2

x∈∗

where ∗ ⊆ −  is some finite interval, Fh x = loss of generality, one can take ∗ = 0 1 .



−x



∗

(4.28) fˆh tdt. Without

Theorem 3 Let X n = X1      Xn be i.i.d. r.v.s with a PDF fx that is supported on ∗ = 0 1 . We assume that for x ∈ R, Kx, is continuous, positive, vanishes outside the interval −1 1 , and satisfies  sup Kx ≤ C <  Kxdx = 1 R

x

Then any solution h∗ = h∗ n of (4.28) obeys the condition h∗ → 0

as

n → 

Theorem 4 Let the PDF fx be estimated by the nonvariable bandwidth kernel estimator (4.24). Let fx be located on a finite interval ∗ . Assume that the

142

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

conditions on Kx given in Theorem 3 hold. In addition, we assume that Kx is of second order, fx has two continuous derivatives f ′ x, f ′′ x, and 2 ≥ f ′′ x

f ′ x ≥ 1 > 0

for x ∈ ∗ 

(4.29)

where 1 and 2 are constants such that 2 ≥ 1 C/2 + 1/4. Then any solution h∗ = h∗ n of equation (4.28) obeys the condition    1 − P1 n−/2 ≤ h∗ ≤ 2 n−/2 ≤ 2 exp −n1−2 / 22C + 12  (4.30)

where 1 = 2 K1 −1/2 , 2 = 2 2C + 1K1 1 −1/2 are constants.   Remark 3 The PDF fx = A 2 − x + 012 , x ∈ 0 1 , gives an example of a which satisfies the condition (4.29). Here, A is a normalizing constant giving PDF 1 fxdx = 1. In fact, f ′′ x =  − 2A ≤ 2A = 2 , f ′ x =  − 2Ax + 01 ≥ 0 A/5 = 1 > 0. For Epanechnikov’s kernel Kx = 43 1 − x2 1x ≤ 1 , we have Kx ≤ 43 = C. Then, 2 /1 = 10 ≥ C/2 + 1/4 = 5/8. Theorem 5 Let the PDF fx be estimated by the kernel estimator (4.24). We assume that the conditions on fx and Kx given in Theorem 4 hold and  = 2/5 in (4.28). Then for any solution h∗ of (4.28), we have   P lim n4/5 MSEfˆh∗  ≤ c∗ = 1 n→

∗

where c is some constant that is independent of the sample size n.

Remark 4 It is assumed that fx is located on a finite interval ∗ . This implies that, according to (4.14) or (4.15), one requires a preliminary data transformation in order to retain the results of theorems for the heavy-tailed case.

4.8 4.8.1

The D method for a variable bandwidth kernel estimator Method and results

We consider the estimator  f A t  h1  h defined by (3.15). Simple calculus shows that the bandwidth h which minimizes the corresponding MSE (3.16) at x is given by hx = Dn−1/9 , where D > 0 depends on the unknown PDF fx, on d/dx4 1/fx, and on the kernel Kx. The estimation of the fourth derivative of the inverse PDF is an awkward problem in itself. To avoid it, we shall consider the data-dependent discrepancy method again (Markovich, 2006a). Let h∗ be a solution of the equation A sup Fn x − Fhh x = n−1/2  1

− n−1/m+1 < exp −2n1−2/  (4.32)   where  = 21 + A/G1/m+1 is a constant, G = 1/m + 1! supx  − f m x − m+1 hy y Kydy 0 < < 1, for any  > 2. Remark 5 Pareto (4.9), exponential, and normal PDFs are examples of PDFs that satisfy the conditions of Theorem 7. Let ℜ be a compact set in R. Given  > 0, we use the following notation of Hall and Marron (1988): ℜ ≡ x ∈ R  for some y ∈ ℜ x − y ≤   where  ·  is the usual Euclidean norm.

Theorem 8 Let fx and 1/fx have four continuous derivatives and fx be bounded away from zero on ℜ . Let the PDF fx be estimated by a variable bandwidth kernel estimate  f A xh1  h as in (3.15). Assume that the conditions on Kx given in Theorem 7 hold for m = 3. Furthermore, assume that Kx is symmetric, has two bounded derivatives, and vanishes outside a compact set. Assume that the nonrandom bandwidth h1 in (3.15) obeys h1 = c∗ n−1/5 , where c∗ > 0 is some constant. Then, for any solution h∗ of (4.31), we have     P lim n4/9 E f A xh1  h∗  − fx ≤ x = 1 n→

where x = K3 /24 d/dx4 1/fx4 , and  is defined in Theorem 7.

Corollary 1 Assume that the conditions of Theorem 8 hold. Assume that EZ · fˆ A xh = 0, where Z is a standard normal r.v. Then MSE f A xh1  h∗  −8/9 may reach order n if a maximal solution of (4.31) h∗ has order n−1/9 .

144

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION 1

1

0.1

f(x)

f(x)

0.01 0.5

1.10–3 1.10–4 0

0

1

2

3

1.10–5

0

40

20

x

60

x

Figure 4.4 Retransformed standard kernel estimate (solid line with circles) and variable bandwidth kernel estimate without transformation (dotted line) with Epanechnikov’s kernel for the Pareto distribution with shape parameter equal to 1 (solid line): body (left) and tail (right). For both estimates h is selected by the D method (h = 021 for the first estimate and h = 011 for the second) and h1 = 1915 is calculated by (2.30).

Remark 6 Since the function of the r.v. X1 (i.e., one term in the sum fˆ A xh) and the normally distributed r.v. Z are independent, the condition EZ · fˆ A xh = 0 is not rigorous. Remark 7 In Theorem 8 we assume that Kx has a compact support. This assumption is not reliable if tail estimation is the object of interest. In this case, a data transformation is required. By way of an illustrative example, Figure 4.4 shows the prevalence of the retransformed kernel estimate in the tail domain in comparison with the variable bandwidth kernel estimate without a preliminary transformation. The adaptive transformation (4.11) is used. One can observe the truncation of estimate (3.15) beyond the sample maximum (Xmax = 133).

4.8.2

Application to Web traffic characteristics

We apply the kernel estimators (3.15) and (4.24) to the Web data, where h is estimated by the discrepancy method (2.42); see Markovich (2006a). These data have already been described in Table 1.4. To simplify the calculation the data were scaled, i.e. all values were divided by the scaling parameter s. To check whether the measurements corresponding to the s.s.s., s.r., d.s.s. and i.r.t. samples are derived from heavy-tailed distributions, we have estimated the EVI  by Hill’s method (1.5). In Table 4.1 one can see the Hill estimates ˆ =   H n k. The numbers of retained data k for all data sets are selected by the bootstrap method ˆ one may conclude that (Markovich, 2005a; see also pp. 36, 37 above). Observing ,

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

145

Table 4.1 Estimation of the EVI and the bandwidth for Web traffic characteristics. r.v. s.s.s. s.r. i.r.t. d.s.s.

ˆ

k

h1

hs

hv

101 − Tˆ Xn 

0949 0898 0712 0601

50 211 211 50

0059 0020 0042 0170

0155 0059 0110 1000

0320 0175 0250 1100

0382 075 0519 0063

Reprinted from Proceedings of 2nd Conference on Next Generation Internet Design and Engineering, Valencia, Estimation of heavy-tailed density functions with application to WWW-traffic, Markovich NM, Table 2, © 2006 IEEE. With permission from IEEE.

the estimates of the tail index  = 1/ are always less than 2 for all data sets considered. It follows from the extreme value theory (Embrechts et al., 1997) that at least the th moments,  ≥ 2, of the distribution of s.s.s., d.s.s., s.r., i.r.t. may be not finite if we believe that the distribution has a regularly varying tail. The positive sign of ˆ indicates that distributions of the Web traffic characteristics considered are heavy-tailed.

1.10–3

1.10–6 1.10–7

1.10–4

1.10–8

f(X)

f(X)

1.10–9 1.10–10

1.10–5

1.10–11 1.10–6

1.10–12 1.10–13 1.10–14 0.01

0.1

1

X/S

10

100

1.10–7 0.01

0.1

1 X/S

10

100

Figure 4.5 The retransformed standard kernel estimate (4.24) (solid line) and variable bandwidth estimate (3.15) (dotted line) for the s.s.s. (left) and d.s.s. (right) data sets . For both estimates the bandwidth h is selected by discrepancy method (2.42). The data transformation (4.11) is used. The curves nearly coincide for d.s.s. Reprinted from Proceedings of 2nd Conference on Next Generation Internet Design and Engineering, Valencia, Estimation of heavy-tailed density functions with application to WWW-traffic, Markovich NM, Figure 1, (c) 2006 IEEE. With permission from IEEE.

146

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

Hence, we may use (4.11) to transform the data. The PDF g0 x of a new r.v. has been estimated by (4.24) and (3.15) with Epanechnikov’s kernel. The retransformed estimate of the unknown PDF fx was calculated by (3.26): ˆ ˆ ˆ −1/2 ˆ −1/2−1 fˆ x = 05gˆ0 1 − 1 + x 1 + x 

The bandwidths hs and hv in Table 4.1 have been selected by the discrepancy method (2.42) and correspond to estimates (4.24) and (3.15), respectively. The value h1 of the nonvariable kernel estimate fˆh1 x in (3.15) is calculated by (2.30). For Epanechnikov’s kernel we get RK = 3/5, 2 K = 1/5. This formula provides the minimal upper bound of the theoretical value of h that corresponds to the optimal MSE ∼ n−4/5 of estimate (4.24). The retransformed kernel estimates (4.24) and (3.15) have been calculated for the d.s.s. and s.s.s., s.r. and i.r.t. samples (see Figures 4.5 and 4.6). The estimate fx = gx/s/s is shown, where gx/s is the retransformed estimate constructed from scaled data. A logarithmic scale is used for both the X- and Y -axes. The curves of the retransformed kernel estimate (4.24) corresponding to all data sets apart from d.s.s. and of the retransformed kernel estimate (3.15) for the sample s.r. are truncated for large values of x/s because the kernel is not wide enough. Such boundary effects are typical of kernel estimates that are used for compactly 1.10–4

0.01

1.10–5 1.10–3

1.10–6 1.10–7

1.10–4 1.10–5

1.10–9

f(X)

f(X)

1.10–8

1.10–10

1.10–6

1.10–11 1.10–7

1.10–12 1.10–13

1.10–8

1.10–14 1.10–15 0.01

0.1

1 X/S

10

100

1.10–9 0.01

0.1

1

10

100

X/S

Figure 4.6 The retransformed standard kernel estimate (4.24) (solid line) and variable bandwidth estimate (3.15) (dotted line) for the s.r. (left) and i.r.t. (right) data sets . For both estimates the bandwidth h is selected by discrepancy method (2.42). Data transformation (4.11) is used. Reprinted from Proceedings of 2nd Conference on Next Generation Internet Design and Engineering, Valencia, Estimation of heavy-tailed density functions with application to WWW-traffic, Markovich NM, Figure 2, (c) 2006 IEEE. With permission from IEEE.

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

147

supported PDFs. In this case, the kernel estimate of the PDF g0 x located on [0,1] may be equal to zero at the neighborhood of 1 beyond the maximum observation of the sample. This reflects on the retransformed estimate. It becomes equal to zero in the tail and the logarithms of these values go to −. In Maiboroda and Markovich (2004) it was shown that the choice h = 101 − Tˆ Xn  in the neighborhood of 1, where Tˆ x is the transformation (4.11) and Xn is the maximal observation in the sample, may improve the boundary problems. One can compare the values of hs , hv and 101 − Tˆ Xn  in Table 4.1. Obviously, the discrepancy method selects larger values h that are closer to 101 − Tˆ Xn , for estimate (3.15) than for estimate (4.24). Hence, the retransformed variable bandwidth estimate provides a better estimation of the PDF at the tail domain for Web traffic characteristics.

4.9

The 2 method for the projection estimator

Let us consider the projection estimator (2.14). The selection of the smoothing parameter  of this estimator was studied in Vapnik et al. (1992). The latter parameter plays the same role as the bandwidth h of kernel estimates. Theorem 9 Let X1      Xn be a sample of i.i.d. r.v.s with PDF fx ∈ ℘.7 If we take  = n−1/2k+2 in (2.14), then the asymptotic rate of convergence of the estimates fˆpr x X n  to fx is given by the expressions   nk+1/2/2k+2 ˆ pr P lim f x X n  − fx ≤ c = 1 √ n→ ln n   k+1/2/2k+2 Efˆpr x X n  − fx ≤ c = 1 P lim n n→

where c is a quantity that is independent of n. By  ·  we mean the L2 -norm, while by E we mean the expectation with respect to the measure fx. The proof is given in Appendix B. We note that the convergence rate, indicated in the second relation, coincides ˇ with the maximal attainable rate in the class ℘ according to Cencov (1982). Experience shows that for samples of moderate size n the selection of  derived from Theorem 9 leads to unsatisfactory PDF estimates. Assume now that  in (2.14) is selected by the variant of the 2 method (see Section 2.2.4) which consists of the following steps:

7

See Example 5 (Section 2.1) for the definition of class ℘.

148

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

1. If the inequality

   aj n  ≥ 2 2 j=1 j

is satisfied, then  is obtained from the equality  1 Fn x − F  x2 fˆ xdx =   ˆ 2n = n where F  x = statistic

x 0

and

0

(4.33)

(4.34)

ˆ 2n is the estimator of the von Mises–Smirnov fˆpr t X n dt,  2n = n



Fn x − Fx2 fxdx

ˆ pr ˆf x = f x 

if fˆpr x ≥  if fˆpr x < 

(4.35)

 > 0 is some arbitrary constant and  is the mode of the distribution 2n . Such a  need not be unique. 2. If inequality (4.33) is not satisfied, then we take  = n−1/2k+2  Theorem 10 Let X1      Xn be a sample of i.i.d. r.v.s with PDF fx ∈ ℘. If (4.33) is satisfied, then for a sufficiently large sample size, there exists, with probability arbitrarily close to unity, at least one  satisfying (4.34). It is contained in the interval G/n1/2k+3  , where G = Gf is some constant. If there exist various  satisfying (4.34), then for their maximum max we have the inequality Pmax < G/n1/2k+3 < 3n−9/8  Theorem 11 Let X1      Xn be a sample of i.i.d. r.v.s with PDF fx ∈ ℘. If the regularization parameter  of the PDF estimate fˆpr x X n  is obtained by the 2 method, then we have the equality   P lim nk+1/2/2k+3 fˆpr x X n  − fx ≤ c = 1 n→

where c is a quantity that is independent of n.

Remark 8 Although this rate of convergence is smaller than the maximum possible in the class ℘, in practice the 2 method enables us to obtain reliable results for a wide range of fx, even from rather moderate samples. Remark 9 The class ℘ contains the PDFs with compact support [0,1]. In order to apply the projection estimator (2.14) and the 2 method to a heavy-tailed PDF, one has to transform the data to a compact interval by means of some transformation function Tx.

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

4.10

149

Exercises

1. Retransformed kernel estimates. Generate X n according to some heavy-tailed distribution (e.g., Pareto, Weibull with shape parameter less than 1, lognormal), or take heavy-tailed real data. Estimate the shape parameter  by Hill’s method (1.5). Transform the sample X n into a new one Y n by the transformations Tx = ln x, Tx = ˆ ˆ −1/2 2/ arctan x and Tˆ x = 1 − 1 + x (Yi = TXi , i = 1    n), where ˆ is the estimate of the EVI . Calculate the standard kernel estimate gˆ h x by formula (4.24). Take h = n−1/5 , where  is an empirical standard deviation calculated from the sample Y n and Kx = 43 1 − x2 1x ≤ 1 . Calculate the PDF fˆh x of the initial r.v. X1 by the formula fˆh x = gˆ h TxT ′ x

(4.36)

For generated data, compare the retransformed estimates for different heavytailed PDFs using the loss functions in the metric spaces L1 , L2 and C:  1   ˆgh x − gxdx fˆh x − fxdx = !1 = 2

! = !3 =



− 

−

0

2

fˆh x − fx dx

sup fˆh Xi  − fXi 

i=1    n

where fn x, gn x are the estimates of the PDF and f0 x, g0 x are the exact models of the PDF arising from the initial and the transformed r.v.. For each sample size n = 50 100, and 300 construct l = 25 realizations. Calculate the statistics l l  2 1  1 !ij  j2 = !ij − j  j = l = 25 j = 1 2 3 l i=1 l − 1 i=1

2. Repeat Exercise 1 with polygram (2.29) instead of the kernel estimate. Exclude the transformation Tx = ln x from consideration since it leads to an infinite interval − .

3. Comparison of smoothing methods for retransformed kernel estimates. Generate X n according to some heavy-tailed distribution or take heavytailed real data. Transform the sample X n to Y n by the transformation Tˆ x = ˆ ˆ −1/2 1 − 1 + x . Using Y n , calculate a kernel estimate gˆ h x by (4.24) ˆ and then fh x by (4.36). Take Epanechnikov’s kernel function Kx and h = n−1/5 as in Exercise 1. Also, find h of the estimate gˆ h x as a solution of the discrepancy equations 2 n   h Yi  − i − 05 + 1 = 005 2 method G n 12n i=1

150

TRANSFORMATIONS AND HEAVY-TAILED DENSITY ESTIMATION

√

ˆn= nD

√

ˆ n+  D ˆ n−  = 05 n maxD

D method

where     √ + √ √ − √ i  i−1  ˆ ˆ − Gh Yi    nDn = n max nDn = n max Gh Yi  − 1≤i≤n n 1≤i≤n n  h x = x gˆ h tdt, and Y1 ≤ Y2 ≤    ≤ Yn are the order statistics. G − For the generated data, draw plots of fˆh x and compare the selection of h by the D and 2 methods and h = n−1/5 .

4. Repeat Exercise 3 with variable bandwidth kernel estimate (3.15) as the estimate of the PDF of a new r.v. Y1 . Use kernel estimate (4.24) as a pilot estimate fˆh1 x in (3.15). Calculate the value h1 by the formula n−1/5 , where  is an empirical standard deviation constructed from the sample Y n . 5. Boundary kernels. Generate 100 Fréchet distributed r.v.s with DF   Fx = exp −x−1/ 1x > 0 

 = 1

Calculate Hill’s estimate ˆ of . Transform the sample X n to a new ˆ ˆ −1/2 one Y n by the transformation Tˆ x = 1 − 1 + x (Yi = TXi , i = 1     n). Estimate the PDF gx, x ∈ 0 1 of the new r.v. Y1 by kernel estimator (4.24). 3 Select Epanechnikov’s kernel Kx = 1 − x2 , x ≤ 1 for the interval 4 0 Yn . Select h as n−1/5 , where  is an empirical standard deviation constructed from the sample Y n or calculate h bythe 2 method.  Select the kernel Ky = 1/hB2 y − Yn /h for the boundary domain Yn  1 . Here, the kernel B2 is determined by formulas (4.18), (4.19), where Kx is Epanechnikov’s kernel. Use h = 1 − Yn for this case. Find the estimate of X1 by the inverse transform formula (4.36). Compare the estimate with the true PDF in the boundary domain.

5

Classification and retransformed density estimates

In this chapter, the retransformed density estimates are applied to the classification problem. The Bayesian classification algorithm is considered. The retransformed kernel and polygram estimators are used to estimate heavy-tailed PDFs of each class. A new criterion for the quality of a density estimation is implemented. The classifiers obtained are compared in a simulation study. Possible applications of this classification technique to Web traffic data analysis and Web prefetching schemes are considered.

5.1

Classification and quality of density estimation

From the practical point of view, the ability of a PDF estimate to solve a specific problem is a much more important characteristic than its deviation from the true PDF in some functional space. Here, we wish to compare the PDF estimates in terms of the probability of classification (pattern recognition) error. This is a measure of the quality of the classifiers that are constructed by means of these PDF estimates (Maiboroda and Markovich, 2004, p. 579).

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

152

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

We assume that the observed object O belongs to one of M different populations Pk , k = 1     M. The true population is unknown. Let pk = PO ∈ Pk  be the a priori probability that O belongs to a population Pk . We observe some random characteristic (feature) X ∈ R+ of O. For O ∈ Pk we denote the PDF of X by fk x. Our aim is to estimate the type k of the population that O belongs to, from the observation X. The solution of this problem is given by a classifier, which is a function  R+ → 1     M which assigns the estimated type k ∈ 1     M of the population to a value of the characteristic X = x. We suppose that the penalty for making a mistake depends on the type of the true population k and the value of the characteristic x, and denote it by qk x. Then the probability of a misclassification is given by the mean of the loss  = E

M 

1O ∈ Pk  X = kqk X =

k=1

M 

k=1

pk



X=k

qk xfk xdx

(5.1)

The smallest probability of a misclassification is attained for the Bayesian classifier (Devroye and Györfi, 1985) B x = k

if pk qk xfk x ≥ pi qi xfi x for all i = k i k ∈ 1     M

Usually, the true fk x and pk are unknown, but there are some consistent estimates fˆk x, pˆ k available. Then the empirical Bayesian classifier is used: EB x = k

if pˆ k qk xfˆk x ≥ pˆ i qi xfˆi x for all i = k i k ∈ 1     M

It is obvious that EB  ≥ B . Moreover, EB  defined by (5.1) is a r.v. since EB x is a random function depending on those data from which the estimates fˆk x were evaluated. The penalties qi x are defined by the loss caused by misclassification in a real classification problem. If qk x = 1 for all k, one obtains the probability of misclassification as measure of the risk. Since observations in the tail domain are rare, the improvement of the classification in the tail provides a negligible decrease in the probability of misclassification. The effect of accurate tail classification is only significant if the misclassification of tail observations is more dangerous than errors in the body domain. For example, the classification at outliers (large claims or huge files) could be important for insurance or Web traffic analysis, respectively. Therefore, we consider such qk x as are larger in the tail and smaller in the body, that is, qk x →  as x → . At the same time, the Bayesian approach can be applied only if the condition   qk xfk xdx <  (5.2) 0

holds. Penalties qk x should be determined such that the Bayesian classifier is switched from one population to another in the tail region, that is, qi xfi x = qj xfj x, i = j, for sufficiently large x in order to avoid the dominance of certain

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

153

10

q(x)f(x)

1

0.1

0.01 1.10–3

0

2

4

6

8 x

10

12

14

16

Figure 5.1 Selection of penalty functions for two Pareto PDFs (4.9) with parameters

1 = 03 and 2 = 12: q1 x 1 x (solid line), q2 x 2 x (dotted line).

qi xfi x over others at tail points. Otherwise, the Bayesian classifier assigns these points to the same class and an enhanced PDF estimation accuracy has almost no effect. Figure 5.1 shows an√example of such a selection for two populations. Here √ q1 x = x + 10, q2 x = x, and p1 = p2 = 05. With regard to PDF estimation, classification means that we estimate M different PDFs. Then we use the estimates in the empirical Bayesian classifier for objects associated with these PDFs. That estimate is the best one which provides minimal probabilistic classification error. However, the classifier EB is not sensitive to the quality of the PDF estimates; for example, PDF estimates of different accuracy that assign the objects to the same class may have the same value EB . Therefore, as a criterion for the quality of a PDF estimate, we can use the estimate of EB  given by ˜ EB  =

M  i=1

pˆ i



 0

qi xfˆi xdx −



 0

max pˆ i qi xfˆi xdx i

(Markovich, 2002). Here ˜ EB  is the probability of a misclassification by the Bayesian classifier of the true PDF estimates fˆi x and the a priori probabilities pˆ i , i = 1  M, of classes. Therefore, ˜ EB  fluctuates near B . The greater the accuracy of the PDF estimate, the closer is ˜ EB  to B . Hence,   M     ˜ EB  − B  =  pˆ i qi xfˆi x − pi qi xfi x dx  i=1 0     max pi qi xfi x − max pˆ i qi xfˆi xdx + i

0

≤

M    i=1

 

0

i

    qi x pˆ i fˆi x − pi fi x dx

154

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

    qi x pi fi x − pˆ i fˆi x dx i 0        qi x pˆ i fˆi x − pi fi x dx  ≤M + 1 max    + max 



i

0

(5.3)

Let T R1 → 0 1 be a strictly monotonically increasing one-to-one transformation. The inverse transformation T −1 and derivatives T ′  T −1 ′ are assumed to be continuous. Let gi y be the PDF of the transformed r.v. Y = TX if O ∈ Pi , and Gi y and Gin y be the corresponding DF and empirical DF, respectively. By virtue of (3.26), we have        1     −1 ˆ qi x pi fi x − pˆ i fi x dx =  qi T xpi gi x − pˆ i gˆ i xdx (5.4)  0

0

≤

+





1

0

0

1

qi T −1 xpi gi x − Epˆ i gˆ i xdx

qi T −1 xEpˆ i gˆ i x − pˆ i gˆ i xdx

Thus, the deviation of ˜ EB  from B  is dependent on the accuracy of the PDF estimation of the transformed r.v. in L1 .

5.2

Convergence of the estimated probability of misclassification

We consider the rate of convergence of ˜ EB  to B . We make the following assumptions: 1. All the r.v.s are positive. 2. The penalties qk x satisfy, in addition to (5.2), the condition   qi xdTx <  0

(5.5)

These conditions are satisfied if, for example, qi x = di + x  fi x ≃ ˆ ˆ −1 ˆ −1+1/  ˆ −2  (1+ x−1+1/   fˆi x ≃ 1 + x , Tx = 1 − 1 + x , di > 0 is a −1 constant, and 0 <  < 2  . The following theorems are proved in Markovich (2002); the proofs are reproduced in Appendix C. Theorem 12 Let X1   Xn be i.i.d. r.v.s with PDF fx. The transformation (4.11) to a triangular PDF gx = 21 − x1x ∈ 0 1 is considered. If the PDF of the transformed r.v. Y = TX is estimated by the kernel estimate n x−x  1  i gˆ x =  K nh i=1 h

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

155

where Kx ≤ c, Kx= 0 for x ≤1, and Kx = 0 otherwise, Kh x = 1 h 1/hKx/h1x ≤ h, −h Kh xdx = −1 Kudu = 1 for 0 < h < 1, and the bandwidth is h = n−   > 0, 0 < d < min0 5 , then  P lim nd ˜ EB  − B  ≤ c1 = 1 n→

where c1 is a positive constant independent of n.

Theorem 13 Let X1   Xn be i.i.d. r.v.s with PDF fx. If the PDF gx of the transformed r.v. Y = TX is such that gx ≤ c <  g ′ x <  for x gx > 0, and is estimated by the polygram (2.29), where m = n+1 , L = n , 0 <  < 1 is the L  smoothing parameter, and 0 < d < 2 , then  P lim nd ˜ EB  − B  ≤ c1 = 1 n→

where c1 is a positive constant independent of n.

Remark 10 Polygram and kernel estimates have identical asymptotic convergence rates that are no worse than n−d , where 0 < d < 05.

5.3

Simulation study

The proximity of ˜ EB  (and EB ) to B  is compared for the kernel estimate and a polygram (2.29). The fixed transformation Tx = 2/ arctanx and adaptive transformation T x (4.11) are applied to both estimates. Pairs of populations with PDFs f1 x and f2 x are used. For example, the latter may be the populations of file sizes of HTML and streaming video connections arising from Web sessions. The mean loss due to misclassification for two classes is given by (Maiboroda and Markovich, 2004, pp. 580–582)     q2 xf2 x1x = 1dx q1 xf1 x1x = 2dx + p2  = p1 0

0

Here  R+ → 1 2 is the classifier. Since the exact Bayesian risk of the misclassification B  (the best possible) can be computed for known PDFs, the relative accuracy of the classifier EB x and a PDF estimate is taken equal to

1/m m i=1 i EB  Jc EB  = − 1 B  and

 

 1/m m  ˜ EB  i=1 i   − 1  Jd EB  =    B

where ˜ EB  = pˆ 1



 0

q1 xfˆ1 x1EB x = 2dx + pˆ 2





0

q2 xfˆ2 x1EB x = 1dx

156

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

The mean is taken over all m samples of the Monte Carlo study. The worse the PDF estimate, the larger the value Jd EB  is. The larger Jc EB  is, the worse the classifier EB x is. Investigation of the quality of the classifiers Using the adaptive transformation T x, the performance of an empirical Bayesian classifier can be improved only if • the estimator of the EVI ˆ n is sufficiently accurate (for Hill’s estimator this means that the EVI and the sample size should be large enough); • the EVIs of the compared PDFs differ noticeably; • the misclassification losses qi x are significantly high at the tail domain (in comparison to the losses in the body domain). We generated samples of the known PDFs of a Pareto distribution (4.9) with EVI ∈ 1 3fp  and a Fréchet distribution with DF (1.32), with ∈ 1 2ff . We also considered Pareto(2)–Fréchet(0.3) mixtures PDFs fpf . Here p1 = p2 = 05 were taken as proportions of the classes. We used the penalty functions qk x = x1/4 , which ensure the convergence of (5.2). In Table 5.1, Jc EB  is shown for two types of transformed polygram (Pl and Plf ) and transformed kernel estimates with Epanechnikov’s kernel (Ke and Kef ). Pl and Ke are calculated using the transformation T x, but Plf and Kef are calculated using the fixed transformation Tx. The ratios of Jc EB  for the polygram and kernel estimates and both transformations are given in the last two columns. A smoothing parameter is calculated by the formula h = n−1/5

(5.6)

( is the standard deviation of the transformed data), which performs well with Epanechnikov’s kernel; see Devroye and Györfi (1985). The number of quantile intervals in the polygram was chosen as n/10 for the sample size n ∈ 50 100 300 500. Here r denotes the integer part of r. The number of Monte Carlo repetitions is given by m = 1000. We used Hill’s estimate for the EVI with the smoothing parameter k = 01n. 1 The Pareto–Fréchet mixture was investigated in more detail. Namely, for fpf 2 the integral in the measure EB  is taken over 0  and for fpf over 6 , i.e., only in the tail domain. For ease of understanding, the results of Table 5.1 are presented in Figures 5.2 and 5.3. The simulation study shows the following: • The adaptive transformation (4.11) always improves the quality of the classification of the kernel estimate (in comparison to the fixed arctan transformation), but makes the polygram worse for relatively small samples.

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

157

• The superiority of T -transformed estimates becomes more evident if the sample size increases. • A kernel estimate is better than a polygram if the classification on the unrestricted domain is considered. However, if the tail domain classification is significant, then a smoothed polygram is preferable for relatively small samples.

Investigation of the quality of the PDF estimates Samples with known PDFs were generated: Pareto PDFs (4.9) fp  with EVI

= 03 12, Burr PDFs fb  with parameters = 1  = −1 and = 4  = −2, fx = x−1 1 + x −−1 , where the EVI and parameter  are defined by

= 1/ and  = −1/, respectively. Mixed pairs of distributions Burr1 −1– Pareto(1.2) fpb  were also studied. Penalty functions satisfying (5.2) were used, namely q1 x = d1 + x and q2 x = d2 + x , where d1  d2 > 0, and Table 5.1 Loss due to misclassification. PDF

Jc EB  · 103

n Pl

Plf

Ke

Kef

Pl/ Plf

Ke/ Kef

fp

50 100 300 500

131 54 47 39

62 51 46 38

57 41 22 11

69 52 36 26

211 106 102 103

082 079 061 042

ff

50 100 300 500

153 65 43 23

135 58 41 25

142 55 18 9

168 69 26 20

113 112 105 092

085 080 069 045

1 fpf

50 100 300 500

347 321 257 215

485 467 410 379

72 31 8 6

77 43 13 12

072 069 063 057

093 072 062 050

2 fpf

50 100 300 500

41 10 9 4

32 9 15 7

55 27 9 5

73 51 25 21

128 111 06 057

075 053 036 024

Reprinted from Computational Statistics, 19(4), pp. 569–592, Estimation of heavy-tailed probability density function with application to Web data, Maiboroda RE and Markovich NM, Table 1, © 2004 Physica-Verlag, A Springer Company. With kind permission of Springer Science and Business Media.

158

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES 200

150

150 J(n)

J(n)

100 100

50 50

0

0

100

200

300

400

0

500

0

100

200

300

400

500

n

n

Figure 5.2 Jc EB  · 103 over 0  for estimates Pl (solid line), Plf (dotted line), Ke (solid line with + marks), Kef (dot-dashed line): (left) Pareto(1)–Pareto(3) PDFs and (right) Fréchet(1)–Fréchet(2) PDFs. Reprinted from Computational Statistics, 19(4), pp. 569– 592, Estimation of heavy-tailed probability density function with application to Web data, Maiboroda RE and Markovich NM, Figure 2, © 2004 Physica-Verlag, A Springer Company. With kind permission of Springer Science and Business Media.

80

600

60 J(n)

J(n)

400 40

200 20

0

0

100

200

300 n

400

500

0

0

100

200

300

400

500

n

Figure 5.3 Jc EB  · 103 for Pareto(2)–Fréchet(0.3) PDFs for estimates Pl (solid line), Plf (dotted line), Ke (solid line with + marks), Kef (dot-dashed line): (left) over 0  and (right) over 6 . Reprinted from Computational Statistics, 19(4), pp. 569–592, Estimation of heavy-tailed probability density function with application to Web data, Maiboroda RE and Markovich NM, Figure 3, © 2004 Physica-Verlag, A Springer Company. With kind permission of Springer Science and Business Media.

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

159

Table 5.2 Quality of the PDF estimates. PDF

Jd EB  · 103

n Pl

Plf

Ke

Kef

Pl/Plf

Ke/Kef

fp

50 100 500

21 34 24

87 46 43

84 11 3.365

86 1 5.029

0241 0739 0558

0.977 11 0.669

fb

50 100 500

45 46 42

29 51 55

41 24 13

63 58 15

1551 0901 0763

0.651 0.414 0.866

fpb

50 100 500

133 102 89

279 209 156

201 121 41

242 186 131

0477 0488 0571

0.831 0.651 0.313

 = 05 min1/ 1  1/ 2 . Here 1 and 2 are the EVI estimates for population pairs. The constants dk for two populations k = 1 2 were chosen from the condition of switching the Bayesian criterion at the 1 − -quantile point of one of the populations, that is, q1 xf1 x and q2 xf2 x intersect in the point x⋆ = F1−1 1− (F1 x is DF of one of the populations). For the generalized Pareto distribution x∗ = − 1 − 1/ 1  The values d1 and d2 were determined by the equations d1 + x∗  f1 x∗  = d2 + x∗  f2 x∗   i.e., d1 = d2 + x∗  f2 x∗ /f1 x∗ 1/ − x∗ , and d2 > x∗ f2 x∗ /f1 x∗ −1/ − 1 ,  was chosen such that d1 and d2 were positive and approximately equal to q1 x and q2 x, respectively. The penalty functions were computed for  = 025. The smoothing parameter h and the number of intervals for the polygram were computed in the same way as before. The EVI estimates were found by the Hill’s estimate, in which the smoothing parameter k was chosen by the bootstrap method (Section 1.2.2). The proportions of classes were taken as p1 = p2 = 05. Table 5.2, based on Table 1 in Markovich (2002), lists Jd EB  for two types of transformed polygrams (Pl and Plf ) and transformed kernel estimates with the Epanechnikov kernel (Ke and Kef ) and the same notation as before. The modeling shows that: • the adaptive transformation (4.11) leads to better estimation than the fixed transformation, though the EVI estimation is rough; • the kernel estimate is worse than the polygram for relatively small samples, but better for large samples.

160

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

5.4

Application of the classification technique to Web data analysis

In this section, some examples of a potential application of the classification procedure to Internet data are provided (Maiboroda and Markovich, 2004, pp. 583–585).

5.4.1

Intelligent browser

When a user gets access to the Web, he generates one or several sessions. A session can consist of several http requests. An http request is generated each time the user clicks on a link. Sometimes, several http requests can be generated at the same time because the browser automatically loads images from a Web page. Let us consider an ‘intelligent browser’ for a wireless environment such as a Universal Mobile Telecommunications System (UMTS) network. This can select which image to load depending on the typical behavior of the user. Specifically, suppose the browser first offers the user the information about the size of a picture. The user can ask the browser to show him a complete picture or decide not to look at this picture at all. We assume that in order to maintain information about user behavior one can observe the work of the user for some fixed period of time. Then one maintains two data sets: the sizes of rejected pictures (i.e. the ones which the user did not want to open) and the sizes of accepted pictures (opened by the user after preliminary information from the browser). Furthermore, the PDFs f1 x and f2 x of both samples can be estimated, for example, by the method described in Section 4.3. Using these PDF estimates one can construct an empirical Bayesian classifier EB x to decide for the user whether the picture should be opened completely for browsing. This is a typical classification problem. The mean penalty of the misclassification for two classes is given by:     EB  = p1 q2 xf2 x1EB x = 1dx q1 xf1 x1EB x = 2dx + p2 0

0

If the classifier has made a mistake and the browser opened the picture (i.e., EB assigns the picture to the second class) which is not useful for the user (i.e., the picture is actually   related to the first class), then the mean loss due to the browser is equal to p1 0 q1 xf1 x1EB x = 2dx. Similarly, if the browser did not open a useful picture because the classifier assigned   it to the first class then the mean loss due to the browser is determined by p2 0 q2 xf2 x1EB x = 1dx. Here, p1 and p2 are the proportions of the pictures related to the first and second class in the common measurement. The penalty functions q1 x and q2 x could be defined as the financial losses of the network. Then EB  reflects the quality of the classification.

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

5.4.2

161

Web data analysis by traffic classification

The http requests may be of different types: ordinary Web pages based on HTML descriptions, advanced Synchronized Multimedia Integration Language (SMIL) presentations with large images, and multimedia streams. We suppose that separate observations of all sources are available, for example, file sizes are measured. Then one can estimate the file size PDFs of the sources and provide the classifier EB x. Furthermore, the classification of any new http request can be given and the best service of the request may be provided.

5.4.3

Web prefetching

Web prefetching aims to reduce the Web user’s perceived latency. First, the user’s preferences and accesses (e.g., favorite objects) are investigated. Then the prefetching engine lists the objects which should be prefetched. The prefetching engine is located in the web browser or in an intermediate web proxy server. As a result, the favorite objects are predownloaded and kept in cache. Mozilla Firefox is an example of a Web browser with the prefetching function (Padmanabhan and Mogul, 1996). Algorithms for predicting user preferences are given, for example, in Davison (2002) and Padmanabhan and Mogul (1996). The idea of the ‘intelligent browser’ can be applied for such prediction. Suppose the user demands one object but the prefetching engine predicts another. Then the prefetching process is interrupted and the user’s request is satisfied. If the user demands the object from the prefetching list then the request is provided to the user with zero service time. The objects from the prefetching list can be considered as the first class, the other objects as the second class.

5.5

Exercises

1. Retransformed kernel estimates and classification. Generate two samples X1n and X2n of size n = 100 with known PDFs of a Pareto distribution (4.9) with 1 = 1 f1 x and 2 = 3 f2 x, respectively. • Estimate f1 x and f2 x by the retransformed kernel estimator with Epanechnikov’s kernel (2.21). Use the adaptive transformation (4.11), where is estimated by Hill’s estimate (1.5). Apply formulas (3.26) and (4.24). • Take p1 = p2 = 05 as a priori proportions of the classes.

• Take as smoothing parameter in (4.24) h = n−1/5 , where  is a standard deviation calculated from new samples T ˆ 1 X1n  and T ˆ 2 X2n . √ √ • Use the penalty functions q1 x = x + 10, q2 x = x.

162

CLASSIFICATION AND RETRANSFORMED DENSITY ESTIMATES

• Mix samples X1n and X2n together and classify the points Xi of the obtained sample by the classifier

1 if p1 q1 xfˆ1 x ≥ p2 q2 xfˆ2 x Xi belongs to class 2 if p1 q1 xfˆ1 x ≤ p2 q2 xfˆ2 x • Calculate the loss function

05q1 xf1 x x = 05q2 xf2 x

if q1 xfˆ1 x < q2 xfˆ2 x otherwise

• Calculate the risk of misclassification using the formula    1 uxu′ xdx xdx = R= 0

0

2

where ux = 1/1−x −1 may be taken for simplicity of calculation.

2. Carry out the classification in Exercise 1 for the penalty functions q1 x = d1 + x and q2 x = d2 + x , where d1  d2 > 0  = 05 min1/ ˆ 1  1/ ˆ 2 ,

ˆ 1 and ˆ 2 are estimates of 1 and 2 . Select the constants d1 and d2 from the switching condition of the Bayesian rule (see Figure 5.1) at the 1 − quantile of the first population x⋆ = F1−1 1 −  (F1 x is the DF of the first population), that is, d1 + x∗  f1 x∗  = d2 + x∗  f2 x∗  where x∗ = − ˆ 1 − 1/ ˆ 1 , and d1 = d2 +x∗  f2 x∗ /f1 x∗ 1/ −x∗ 

 d2 > x∗ f2 x∗ /f1 x∗ −1/ − 1 

Select  ∈ 01 025. Compare the risks of misclassification with Exercise 1.

6

Estimation of high quantiles

This chapter discusses estimators of the high quantiles for heavy-tailed distributions. The relative bias and the mean squared errors as well as confidence intervals of estimates are compared in a Monte Carlo study using simulated r.v.s. The distribution of the logarithm of the ratio of Weissman’s estimate to the true value of the quantile is proved to be asymptotically normal. The same result is obtained for a modification of Weissman’s estimate. An application to WWW traffic data is considered.

6.1

Introduction

Suppose we have a sequence of i.i.d. observations X n = X1  X2      Xn  from an unknown DF Fx. Definition 15 For the continuous DF Fx the quantile x = xp of level 1 − p, p ∈ 0 1, is the solution of the equation 1 − Fx = p

(6.1)

Our aim is to evaluate a high quantile, that is, a quantile corresponding to a probability p close to zero when Fx is heavy-tailed. In practice it is often necessary to evaluate the risk of large but possibly rare losses. The analysis of such risks is important in indicating the thresholds of

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

164

ESTIMATION OF HIGH QUANTILES

parameters in complex multi-component systems such as economic and ecological systems, atomic power stations, the Internet etc. High quantiles are usually located at the boundary or beyond the range of the sample. The classical approach of using the empirical DF Fn x in (6.1) as well as weighted quantile estimators1 is not valid for high quantiles since Fn x = 1 holds for x ≥ Xn . Here Xn denotes the largest order statistic corresponding to X n . The lack of information beyond the range of the sample creates the main problem in the estimation of high quantiles. Since Fn Xn  = 1, for p < 1/n it is impossible to estimate the quantiles without knowledge of the behavior of F at infinity. The main idea behind all estimators for high quantiles is to select first some auxiliary pilot estimate of a quantile inside the range of the sample (one can use one of the order statistics close to the boundary as a pilot estimate) and to move this pilot estimate to the right. For this purpose, special scaling parameters are estimated.2 Obviously, in order to extrapolate the pilot quantile beyond the sample range, one needs to use some model of the tail of the distribution. Such models are not available in many applications. Therefore, the asymptotic tail models (1.1)–(1.3) based on the distribution of the largest order statistic are usually used.

6.2

Estimators of high quantiles

To satisfy (1.1) it is necessary and sufficient, by Proposition 3.3.2 in Embrechts et al. (1997), that lim nF bn + an x = − ln H x

n→

x ∈ R an > 0 bn ∈ R

(6.2)

It is evident from (1.3) and for = 0, that

lim t 1 − Fatx + bt = 1 + x−1/

t→

and 1 − Fu ≈

  1 u − bt −1/ 1+  t at

For the 1 − pth quantile the approximation

k/pn ˆ − 1 ˆ xˆ pd = bn/k + aˆ n/k ˆ

may be used (Dekkers et al., 1989).3 Here, ˆ is an estimate of . 1 Let X1 ≤ X2 ≤    ≤ Xn be the order statistics of the sample. Usually Xk , with k = np + 1 ( a denotes the integer part of a), is used as an estimate of the pth quantile xp . It leads to the estimates that are interpolations of order statistics, e.g., to a weighted average xp = 1 − gXj + gXj+1 , where j = np and g = np − j (Dielman et al., 1994). 2 The same principle can be applied to the estimation of the small quantiles when p is close to one. 3 ˆ Asymptotic normality of xˆ pd has been proved for the specific functions aˆ n/k and bn/k; see de Haan and Rootzén (1993).

ESTIMATION OF HIGH QUANTILES

165

The GPD (1.16) and the Pareto-type tail model (Hall, 1990; Hall and Weissman, 1997)   (6.3) 1 − Fx = cx−1/ 1 + dx− + ox−  

where > 0, > 0, c > 0, − < d < , are often used to model the tail of the distribution Fx. Different estimators of high quantiles (e.g., Dekkers et al., 1989; Beirlant et al., 1999; Weissman, 1978) follow from these assumptions on the tail. The form of the tail can be detected by rough methods of heavy-tailed data analysis such as the QQ plot (Embrechts et al., 1997) or by nonparametric tests (Jureˇcková and Picek, 2001). We shall consider some well-known estimators. In the POT estimator the GPD is used as a distribution of excesses over some high threshold u:   − ˆ p

ˆ POT −1  (6.4) xp = u + 1 − Fn u ˆ where ˆ and ˆ are estimates of parameters of the GPD, and Fn u is the empirical DF evaluated at u (McNeil and Saladin, 1997; McNeil et al., 2005). In Weissman (1978) the estimator   k + 1 ˆ xpw = Xn−k  k = 1     n − 1 (6.5) n + 1p

is obtained from (6.2) for the Pareto tail model, that is, for the first type of tail in (1.2), where X1 ≤ X2 ≤    ≤ Xn are order statistics corresponding to the sample X n . In Markovitch and Krieger (2002a) an estimator was proposed that differs slightly from xpw . To obtain this estimator the idea of combining the estimator (3.4) with (3.1) for a heavy-tailed PDF was applied. This implies that the PDF is approximated by the Pareto-type model f x =

1 −1/ −1 2 −2/ −1 x + x

(6.6)

as x ≥ Xn−k , and by some nonparametric estimator f N t for x < Xn−k such that (3.5) holds. We note that for xp > Xn−k , p = Px > xp  = Px > xp x > Xn−k Px > Xn−k  Px > Xn−k  may be replaced by the number of excesses over the threshold Xn−k , namely, k/n. From (3.1) and (3.4) the estimate   minxX  x n−k 1  f tdt f N t1f N t > 0dt + F x = c  0 minxXn−k 

of the DF Fx follows, where   f t  N1 f t  N > 0dt c  = 0

166

ESTIMATION OF HIGH QUANTILES

may be approximated by

 0

 f t  Ndt (see (3.6)):

−1/ −2/ + Xn−k  c  ≈ 1 + Xn−k

(6.7)

Since

Xn−k

0

f N t1f N t > 0dt = c  −



 Xn−k

f tdt

we have for x > Xn−k by (6.6) that  X  x n−k 1 N N  f tdt F x = f t1f t > 0dt + c  0 Xn−k =1 −

1 x−1/ + x−2/  c 

Let 1 − F x = x−1/ + x−2/  Then we can use Px > xp x > Xn−k  =

Hence, since

1 − Fxp  1 − FXn−k 

      Xn−k 1/ Xn−k 1/ 1− F xp  1 ≃ < 1+  1 − F Xn−k  c  xp xp k p≃ nc 



Xn−k xp

1/

+



Xn−k xp

2/ 

holds approximately, one can expect that the statistic − ˆ   ˆ pnc  c xp = Xn−k −05 + 025 + k

(6.8)

approximates xp . Here, ˆ is an estimate of the EVI that determines the shape of the tail. The estimate xpc differs from Weissman’s estimate xpw in the normalizing multiplier, reflecting the fact that the estimate  F x of the DF Fx includes not only the parametric estimate of the tail domain as in xpw , but also the “body” estimate.

X Since it is assumed that 0 n−k f N tdt = 1, we are able to use any parametric or nonparametric estimate of the PDF “body” with such a property. The common disadvantage of the high quantile estimators is their sensitivity to the choice of threshold. This may be the value of u in the estimator xpPOT or the number of order statistics k in xpc and xpw . An estimate of k is also required to estimate the EVI . One can apply Hill’s estimator (1.5) or other estimators such

ESTIMATION OF HIGH QUANTILES

167

as the moment estimator, the ratio estimator, or the UH estimator that are valid not only for positive (see Section 1.2). When k increases (i.e., Xn−k decreases), the variance of the EVI estimate decreases but the bias increases. However, when k decreases (i.e., fewer data are used) the bias tends to 0 but the variance increases. 2  Theoretically, an optimal value of k has to minimize the MSE E xˆ p k − xp . An exact expression for MSE is not available since xp is unknown. Thus, finding k is usually a matter of minimizing the asymptotic MSE, that is, the asymptotic 2  expectation asE xˆ p k − xp or more precisely its bootstrap estimator (Ferreira 2   et al., 2000) or asE log xˆ p k/xp (Beirlant et al., 1999). In Matthys et al. (2004) it is the estimate of the asymptotic MSE of xpw given in (6.10) with incorporated ML estimates of the unknown parameters , bnk ,  of the distribution that is minimized to find k. The ML method is used in the framework of an exponential regression model for log-spacings of order statistics. The latter method is extended to censored data. −1 Hall and Weissman (1997) proposed to minimize the MSE EF  F 2 p − p , where F  is some tail estimate. A simulation study with heavy-tailed distributions has shown that the quantile estimate xpc is better than xpPOT and xpw for the highest quantiles, and has demonstrated smaller mean squared errors (Markovitch and Krieger, 2002a). The estimate xpPOT is essentially less accurate due to the estimation of both GPD parameters by the ML method in addition to the threshold u. It is shown in Matthys and Beirlant (2001) that the GPD shape parameter and the high quantile estimates are sensitive to the parameter computation method – the ML method (Smith, 1987), the PWM method (Hosking and Wallis, 1987) and the EPM (Castillo et al., 2006). Formally, the threshold u is not random. One estimates the parameters of the GPD for a fixed u. Nevertheless, one can select some quantile of the unknown distribution as a threshold and replace it by an empirical quantile that is random (McNeil and Saladin, 1997). The alternative is to select one of the sample points Xn−k (Beirlant et al., 2004, p. 149).

6.3

Distribution of high quantile estimates 4

Here, we consider the distributions of xpc (6.8) and xpw (6.5). It is evident that xpc xp

=

xp



Xn−k  ˆ ˆ pnc  −05 + 025 + k

4 This section is taken from Performance Evaluation, 62(1–4), pp. 178–192, High quantile estimation for heavy-tailed distributions, Markovich NM, Section 3, © 2005 Elsevier. With permission from Elsevier.

168

ESTIMATION OF HIGH QUANTILES

and log



xpc xp







ˆ pnc  = log Xn−k − log xp − ˆ log −05 + 025 + k   ˆ c  ≃ log Xn−k − log xp − ˆ log  an



where an = k/pn. Note that  c  w xp xp ˆ log ≃ log − ˆ log c  xp xp

(6.9)

It is proved in Markovich (2005b) that the distributions of the logarithms of ratios xpc and xpw to the true value of the quantile xp are asymptotically normal. ˆ Following Dekkers and de Haan (1989), Let us use Hill’s estimate (1.5) as . to derive the asymptotics we require that k/p · n have positive limit as n → . Theorem 14 (Markovich, 2005b) Let the tail distribution be of Pareto type (6.3) and k n →  nk → 0, p = pn ∼ c∗ · nk → 0, c∗ > 0, as n → . Then logxpw /xp  − a w

−→d N0 1

logxpc /xp  − a + k + 1/n  c

−→d N0 1

where a = dc

−



k+1 n



−p







⎛ ⎞   2   2 2 k + 1 k ⎠ ⎝ 1 − dc− + log 2w = k n np         2 k + 1 k + 1 k + 1 2c = 2w + − 2 1 − dc−  k n n n The proof of the theorem is given in Appendix D. Theorem 14 shows that the expectation of the distribution of logxpc /xp  is larger than that of logxpw /xp , while the variance is less. The difference becomes negligible as the sample size increases. Asymptotic normality of the estimate log xpw is also given in Matthys and Beirlant (2003) for the class of heavy-tailed distributions with regularly varying tails, that −1/ is, 1 − Fx =  wx ℓx, where ℓx is a slowly varying function. The asymptotic MSE of log xp /xp ,

ESTIMATION OF HIGH QUANTILES



169

2 

  w 2  xp 2 k+1 = asE log 1 + log xp k+1 n + 1p    2   k+1 k+1  1 1 2 + bnk  log 1− +  pn + 1 1− pn + 1 (6.10) where bnk = b n + 1/k + 1 

(6.11)

is obtained under a specific assumption on ℓx denoted by Rl : There exists a real constant  ≤ 0 and a rate function b satisfying bx → 0 as x → , such that for all  ≥ 1, as x → , logℓx/ℓx ∼ bxk , with k  =  − 1/, which is to be read as log  if  = 0. The distribution (6.3) satisfies assumption Rl  with bx =  dc x 1 + o1

(6.12)

where  = − . Then the result of Theorem 14 can be rewritten in this notation as    w 2   xp k 2 2 asE log 1 + log (6.13) = xp k np     2     1 k+1  2 k+1  1 2 +  1+ + bnk 2 1 +  pn k d cn This is similar to (6.10), apart of the second term in the final brackets. It follows from (6.10)–(6.12) that  cn 2 k+1 1 2 bnk log c∗  (6.14) log ∼− 1− pn + 1 k

We derive from (6.13) that       2 k+1  cn 2 1 2 1 bnk ∼  1+ k d cn k k The latter term goes to zero faster than (6.14).

6.4 6.4.1

Simulation study Comparison of high quantile estimates in terms of relative bias and mean squared error

For the comparison of xpPOT , xpw and xpc we use some of the distributions applied in McNeil and Saladin (1997), namely, the Pareto distribution with 1/ ∈ 1 2 and the Student distribution with 1/ ∈ 1 2. Following McNeil and Saladin (1997),

170

ESTIMATION OF HIGH QUANTILES

we use as characteristics of the quantile estimates xˆ pi ∈ xpPOT  xpw  xpc  the empirical estimates of the bias and the mean squared error of the estimates expressed as proportions of the true value xp : %Bias = %RMSE =

1 NR



NR

ˆ pi i=1 x

1 NR

xp NR

− xp

xpi i=1 ˆ xp



− xp 2



Here NR = 25 is the number of repetitions in our Monte Carlo study. The parameters

and of xpPOT were calculated by the ML method and the threshold u as an empirical quantile of the underlying distribution. The EVI of xpc and xpw was calculated by Hill’s 2 estimate (1.5) and k from the minimum of the bootstrap estimate  of E xˆ p k − xp ,     1  k1  = E xˆ p⋆ n1  k1  − xˆ p n k 2 X n MSEn with respect to k1 . Here, xˆ p⋆ n1  k1  is the estimate of the quantile calculated from the resample of size n1 with parameter k1 . Such a resample is drawn randomly from the sample X n with replacement. For the bootstrap, 25 resamples were used. The values of the auxiliary parameters  and  for the calculation of the size of the resamples between k and k1 were taken equal to  = 2/3, n1 = n and relation k = k1 nn 1 = 1/2, similar to Section 1.2.4. The results of the simulation are shown in Table 6.1. The results for xpPOT are compiled from McNeil and Saladin (1997). The simulation study illustrates that the quantile estimate xpc is better than xpPOT and xpw especially for the highest quantiles, demonstrating the smaller mean squared error results. This conclusion is in agreement with that which follows from Theorem 14.

6.4.2

Comparison of high quantile estimates in terms of confidence intervals

Theorem 14 does not allow us to construct asymptotic confidence intervals, since a, w and c depend on the unknown parameters of the distribution. Here, we describe the nonasymptotic bootstrap confidence intervals considered in Markovich (2005b). One can find more about bootstrap confidence sets in Shao and Tu (1995, Chapter 4). It follows from Theorem 14 that the logarithms of both estimates xpw and xpc have asymptotically normal distributions. However, in order to get better confidence intervals for finite samples one has to assume that the estimates of quantile xˆ p constructed over the set of the samples are normally distributed. The mean and variance of the normal distribution are constructed by these estimates xˆ p1      xˆ pNR ,

ESTIMATION OF HIGH QUANTILES

171

Table 6.1 Simulation results of quantile estimation. Quantile Sample estimate size

%Bias 1 − p = 099

xpPOT xpc xpw xpPOT xpc xpw

250

xpPOT xpc xpw xpPOT xpc xpw

250

500

500

xpPOT xpc xpw xpPOT xpc xpw

250

xpPOT xpc xpw xpPOT xpc xpw

250

xpPOT xpc xpw xpPOT xpc xpw

250

500

500

500

25 −97 −93 258 −97 −73 1052 −275 84 769 −198 120 062 14 1267 −115 1033 1086 112 −152 341 095 −222 177 1475 −174 377 398 −277 209

%RMSE

1 − p = 0999

1 − p = 099

1 − p = 0999

Pareto distribution 2360 −229 −66 1916 −96 104

1/ = 2 2917 180 247 2088 157 194

13218 277 201 9193 174 227

Pareto distribution 20749 −255 165 5170 −250 121

1/ = 1 8328 355 455 5025 329 405

161114 582 939 25402 447 655

Standard lognormal distribution 309 1884 29545 2301 35472 2935 185 1295 23119 1895 26117 2056 Student distribution 461 32 122 322 27 628

1/ = 2 2517 379 524 2072 257 304

Student distribution 1/ = 1 (Cauchy) 13152 6982 −289 404 557 623 3186 4016 −317 366 310 477

5428 34152 40896 3926 25645 30513 8124 698 1579 6516 325 863 49595 577 1327 16380 421 727

Reprinted from Computer Networks, 40(3), pp. 459–474, The estimation of heavy-tailed probability density functions, their mixtures and quantiles, Markovitch NM and Krieger UR, Table 1, © 2002 Elsevier. With permission from Elsevier.

172

ESTIMATION OF HIGH QUANTILES

where NR is the number of samples. Then one can calculate the tolerance limits of the confidence intervals by the well-known formula   meanˆxp  −  · StDevˆxp  meanˆxp  +  · StDevˆxp   (6.15)

where meanˆxp  and StDevˆxp  are the empirical mean and standard deviation of the NR estimates (Smirnov and Dunin-Barkovsky, 1965). As before (pp. 24, 25), such an interval is constructed in such a way that 100 1 − p% part of the distribution falls into it with probability P. The value  is calculated by (1.34)–(1.36). We have  = 1645 for p = 01, which corresponds to a 90% confidence interval. Then  = 1776 holds when NR = 500. For both estimates xpc and xpw , Hill’s estimator (1.5) is used to estimate the EVI . The number of largest order statistics k for the latter estimate and for the order statistic Xn−k in (6.5) and (6.8) is obtained from the minimum of the bootstrap estimate of the mean squared error of the quantile estimation, that is, 2  E xˆ p k − xp = biasˆxp 2 + varianceˆxp . To construct this bootstrap estimate we use resamples of smaller sizes n1 than n (see the bootstrap method in Section 1.2.2 for details). As values of the auxiliary parameters  and for the calculation of the size of a resample n1 = n and the relation k = k1 n/n1  among k and k1 ,  = 2/3 and = 1/2 are selected, as in Markovitch and Krieger (2002a). Then one can find the minimum of the estimate of the MSE,    1  k1  = E xˆ p⋆ n1  k1  − xˆ p n k 2 X n  = b∗ n1  k1 2 + var ∗ n1  k1  MSEn with respect to k1 , where b∗ n1  k1  and var ∗ n1  k1  are the bootstrap estimates of the bias and variance, xˆ p⋆ n1  k1  is the quantile estimate with parameter k1 , constructed from the resample Xn∗1 of the size n1 that is less than n.5 Such resamples are drawn randomly from the sample X n of the size n with replacement. Since the DF Fx is unknown, one has to use instead of b∗ n1  k1  and ∗ var n1  k1  their empirical estimates: B 1 xˆ ⋆b n  k  − xˆ p n k B b=1 p 1 1  2 B B 1  1 ∗ ⋆b ⋆b var n1  k1  = xˆ n  k   xˆ n  k  − B − 1 b=1 p 1 1 B b=1 p 1 1

b∗ n1  k1  =

The means, standard deviations, mean squared errors and 90% confidence intervals of the estimates xpw and xpc are given in Tables 6.2 and 6.3 for different heavy-tailed distributions and NR = 500. A Pareto distribution with DF Fx = 1 − x−1/ , x > 0, and parameter ∈ 1/2 1 and Weibull distribution with  1  k1  the sample X n is fixed and the expectation is calculated under all In the expression for MSEn theoretically possible resamples Xn∗1 .

5

ESTIMATION OF HIGH QUANTILES

173

DF Fx = 1 − sxs−1 exp−xs  and s = 1/ = 05 were investigated. Sample sizes n ∈ 100 1000 were taken. The true values xp of quantiles of level 1 − p are given in Table 6.4. From Tables 6.2 and 6.3 one can conclude that • the bias of the estimate xpw is less than that of xpc , but the variance is larger for xpw ; • the MSE of the estimate xpc tends to be less than that of xpw , especially for smaller sample sizes; • the confidence intervals of xpw are wider than those of xpc ; • the means of both estimates are far away from the true value of a 999% quantile for a Weibull distribution; however, the confidence interval is better for xpc , especially for smaller samples.

Table 6.2 Tolerant 90% confidence intervals of estimates xpw and xpc for heavy-tailed distributions: 500 samples of n = 100 observations each. PDF

xpc

1 − p·100% meanxpc  (StDevxpc )

Pareto =1

99

999

Pareto = 1/2

99

999

Weibull =2

99

999

xpw Confidence interval

meanxpw  (StDevxpw )

Confidence interval

75.814 −7567 (46.949) 159195) MSE = 2789 · 103

117957 −43487 90903 279401 MSE = 8586 · 103

8.259 (2.116)

10.132 4614 (3.107) 1565 MSE = 9671

963.094 −1795 · 103  3 1553 · 10  3721 · 103  MSE = 2413 · 106 4501 12017 MSE = 7509

26.562 3394 (13.045) 4973 MSE = 195786

25.398 −4366 (16.759) 55162 MSE = 298420

182.084 −305835 (274.729) 670003 MSE = 9353 · 104

1616 · 103 −311 · 103  3 2661 · 10  6342 · 103  MSE = 7460 · 106

34.002 −4287 (21.559) 72291 MSE = 470450

38.719 −31206 (39.372) 108644 MSE = 1857 · 103

487.721 −2988 · 103  1957 · 103  3963 · 103  MSE = 4023 · 106

Reprinted from Performance Evaluation, 62(1–4), pp. 178–192, High quantile estimation for heavy-tailed distributions, Markovich NM, Table 1, © 2005 Elsevier. With permission from Elsevier.

174

ESTIMATION OF HIGH QUANTILES

Table 6.3 Tolerant 90% confidence intervals of estimates xpw and xpc for heavy-tailed distributions: 500 samples of n = 1000 observations each. PDF

xpc

1 − p·100% meanxpc  (StDevxpc )

Pareto =1

99

999

Pareto = 1/2

99

999

Weibull =2

99

999

xpw Confidence interval

80.452 51533 (16.272) 109351) MSE = 646902

meanxpw  (StDevxpw )

Confidence interval

10193 61364 22841 142496 MSE = 525436

791.071 274978 (290.593) 1307 MSE = 1281 · 105

1051 · 103 321879 (410.541) 1780 MSE = 1711 · 105

27.733 19258 (4.772) 36208 MSE = 37904

32.146 21293 (6.111) 42999 MSE = 37618

8.879 7346 (0.863) 10412 MSE = 2001

23.424 16213 (4.06) 30635 MSE = 21394

75.566 12836 (35.321) 138296 MSE = 2023 · 103

10.035 8177 (1.046) 11893 MSE = 1095

21.902 14814 (3.991) 2899 MSE = 16410

76.795 9971 (37.626) 143619 MSE = 2261 · 103

Reprinted from Performance Evaluation, 62(1–4), pp. 178–192, High quantile estimation for heavy-tailed distributions, Markovich NM, Table 2, © 2005 Elsevier. With permission from Elsevier.

Table 6.4 True values of high quantiles for different heavy-tailed distributions. PDF Pareto, = 1 Pareto, = 1/2 Weibull, = 2

1 − p · 100%

xp

99 999 99 999 99 999

100 1000 10 31623 21208 47717

Reprinted from Performance Evaluation, 62(1–4), pp. 178-192, High quantile estimation for heavy-tailed distributions, Markovich NM, Table 3, © 2005 Elsevier. With permission from Elsevier.

ESTIMATION OF HIGH QUANTILES

175

The first conclusion coincides with the conclusions of Theorem 14. The last two conclusions may be explained by the larger variance of xpw , especially for smaller sizes.

6.5

Application to Web traffic data6

High quantile estimators may be applied to determine the thresholds of traffic parameters, (Markovich, 2005b). For example, to optimize TCP one can estimate the quantiles of delays between the arrival times of packets and their acknowledgments. We now apply the estimators xpw and xpc to the real Web data described in Table 1.4. Table 6.5 contains the values of the high quantile estimates xpw and xpc for the different characteristics of Web traffic. The EVI of both estimates was estimated by Hill’s estimator, where the parameter k was calculated by the bootstrap method. For this purpose, 150 bootstrap resamples were used. In Table 6.6 the means, standard deviations, and bootstrap confidence intervals of both xpc and xpw for Web traffic characteristics are given. To construct the confidence intervals the procedure described in Section 6.4.2 was used. Here, all estimates were calculated by B = 50 bootstrap resamples from the sample X n with replacement instead of NR samples generated from the known distribution. From (1.34) we have k∗ = 213.

Table 6.5 High quantiles for Web traffic data Quantile estimate

xpc

xpw

r.v.

d.s.s. s.s.s. s.r. i.r.t. d.s.s. s.s.s. s.r. i.r.t.

Quantile value xˆ p · 10−4 1 − p = 099

1 − p = 0999

1.4005 2299 · 103 69.27 0.1445 1.435 2407 · 103 56.67 0.0954

5.812 21 · 104 439.5 0.5493 5.688 202 · 104 431.6 0.5402

Reprinted from Performance Evaluation, 62(1–4), pp. 178–192, High quantile estimation for heavy-tailed distributions, Markovich NM, Table 5, © 2005 Elsevier. With permission from Elsevier.

6

This section is taken from Performance Evaluation, 62(1–4), pp. 178–192, High quantile estimation for heavy-tailed distributions, Markovich NM, Section 5 © 2005 Elsevier. With permission from Elsevier.

176

ESTIMATION OF HIGH QUANTILES

Table 6.6 Tolerant 90% confidence intervals of estimates xpw and xpc for Web traffic data. r.v.

xpc

1 − p·100%

s.s.s.

99 999

d.s.s.

99 999

s.r.

99 999

i.r.t.

99 999

xpw

meanxpc  (StDevxpc )

Confidence interval

meanxpc  (StDevxpc )

Confidence interval

204 · 107 7025 · 106  1312 · 108 9682 · 107  1466 · 104 (4369 · 103 ) 569 · 104 (3253 · 104 ) 6252 · 105 4722 · 104  4196 · 106 7193 · 105  1281 · 103 124765 8068 · 103 3506 · 103 

5437 · 106  3536 · 107  −7503 · 107  3374 · 108  5354 · 103  2397 · 104  −1239 · 104  1262 · 105  5246 · 105  7258 · 105  2664 · 106  5728 · 106  1015 1547 60022 1554 · 104 

2116 · 107 (6296 · 106 ) 2136 · 108 1621 · 108  1424 · 104 (3593 · 103 ) 593 · 104 2978 · 104  5487 · 105 3911 · 104  4119 · 106 8342 · 105  1135 · 103 173505 8026 · 103 (3425 · 103 )

775 · 106  3457 · 107  −1317 · 108  5589 · 108  6587 · 103  2189 · 104  −4131 · 103  1227 · 105  4654 · 105  632 · 105  2342 · 106  5896 · 106  766536 1503 73075 1532 · 104 

Reprinted from Performance Evaluation, 62(1–4), pp. 178–192, High quantile estimation for heavy-tailed distributions, Markovich NM, Table 6, © 2005 Elsevier. With permission from Elsevier.

From the latter study one may conclude that both estimates xpw and xpc give rather similar results since the sizes of the samples considered are large enough.

6.6

Exercises

1. High quantile estimation. Generate X n , n = 100, according to some heavy-tailed distribution (e.g., Burr, Pareto or Fréchet) and determine its true 99% and 999% quantiles. Calculate the 1 − pth quantiles (p = 01, p = 001) by the POT method (6.4). Estimate the parameters ˆ and ˆ by the ML method and method of moments. Take the order statistics Xn−k , k ∈ 10 30 50, as threshold u. Compare the estimates with the true values of the quantiles. Draw conclusions regarding the sensitivity of the POT method to the parameter selection. 2. Calculate the 1 − pth quantiles (p = 01, p = 001) using the estimators (6.5) and (6.8). Calculate ˆ for these estimators using the Hill and moment

ESTIMATION OF HIGH QUANTILES

177

estimators. Select the parameter k of both estimators by the plot and bootstrap methods (Section 1.2.2). Compare the values of high quantiles constructed by these estimates. 3. Calculate the bootstrap confidence intervals for estimates (6.4), (6.5), and (6.8) by formulas (6.15) and (1.34)–(1.36); see Section 6.4.2. For this purpose, calculate estimates xˆ p1      xˆ pB by B = 50 bootstrap resamples with replacement.

7

Nonparametric estimation of the hazard rate function

In this chapter the nonparametric estimation of a hazard rate function is considered for both light- and heavy-tailed distributions. For the heavy-tailed case a transformation approach to the light-tailed case is presented. In the light-tailed case the hazard rate is evaluated as the solution of an integral equation. Such tasks are ill-posed and, hence, the solution is obtained by a statistical analog of Tikhonov’s regularization method. The theoretical background of the latter method and the numerical solution of ill-posed problems using empirical data are presented. The regularized estimates are proved to converge in the uniform metric of space C for a certain choice of the regularization parameter, as well as in the metric of space L2 in the case of a bounded variation of the kth derivative of the hazard rate. Finally, the identification of semi-Markov models and their application in population analysis and teletraffic engineering are discussed. The estimate of the intensity of a nonhomogeneous Poisson process is given. A ratio of hazard rates is considered with regard to the application to the failure time detection in stochastic processes and hormesis detection in biological systems.

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

180

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

7.1

Definition of the hazard rate function

Definition 16 Let X be an r.v., for example the lifetime of some element, with continuous DF Fx, and corresponding PDF fx. The hazard rate function (or, in population analysis, the mortality risk) of X is defined by the function hx =

fx  1 − Fx

(7.1)

The problem of the estimation of the hazard rate function relates to the different behavior of this function on the right-hand side of the real axis. In the case of light-tailed distributions hx →  as x → , while for the exponential distribution hx is equal to the constant intensity  of this distribution, and for heavy-tailed distributions hx → 0 as x → . This is illustrated in Figure 7.1 for the light-tailed normal and exponential distributions, as well as a Weibull distribution with shape parameter 0.3 and the Cauchy distribution which have heavy tails. The von Mises conditions reflect the difference in this behavior for the three classes of a GEV distribution (1.2). The latter implies that Fx is in the domain of attraction of one of the three types. Let the second derivative F ′′ x = f ′ x of Fx exist. Then for  > 0, if ⎧ ′ ⎪  then F converges to Fréchet type ⎨1/ 1 − Fx = −1/ lim then F converges to Weibull type ⎪ x→F fx ⎩ 0 then F converges to Gumbel type

where F is the supremum of the support of Fx; see Reiss (1989, p. 159).

h(x)

4

2

0

0

1

2

3

4

5

x

Figure 7.1 Hazard rate function for standard exponential (horizontal solid line), standard normal (solid line), Weibull with the shape parameter 03 (dashed line), and Cauchy distributions (dotted line).

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

181

The hazard rate function may be defined in terms of the so-called survival function F x by the equation    x htdt  F x = 1 − Fx = P X > x = exp − 0

The survival function determines the probability of living not less than x years. Hence, the hazard rate function determines the probability of the death of an individual in the time interval t t + t, where t is sufficiently small, under the condition of his survival until the age t, that is, P t ≤ T < t + tT ≥ t = ht t + o  t  where T is interpreted as a lifetime (one may rephrase this in terms of the stability of technical systems). In practice, one may use the approximation ht t ≃

dt t + t  Dt

where dt t + t is the number of failed objects observed in the interval t t + t and Dt is the number of elements surviving until age t. The function ht may be interpreted as a rate of transition from the first state to the second in the simplest birth–death model. For instance, if T is the duration of a chronic disease, that is, the time from onset of sickness until death, then the period spent in the first state corresponds to the illness of individuals and the second state corresponds to the death. In this case, ht is the mortality rate among sick individuals (Figure 7.2). Possible applications of the hazard rate function are as follows: • the identification of Markov and semi-Markov models (the estimation of transition rates between different states); • the mortality rate in population analysis; • the ratio between hazard rates of two populations (groups) for failure time detection in stochastic processes, or hormesis detection in biological systems. Many applied problems can be considered in terms of an inverse problem that establishes the relationship between the ‘result’ and the ‘source’ processes. Here, the researcher deals with a model and a set of experimental values of the process under study. The consideration of the ‘result–source’ processes enables

sick

h(t)

dead

Figure 7.2 Two-state survival model.

182

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

one to integrate the mathematical descriptions of various applied problems with the methods to solve inverse problems. This integration is exemplified by the analysis of population processes and the solution of these problems by semi-Markov models, (Markovich, 1995; Markovich and Michalski, 1995). Formal relationships of the probabilities of being in different states1 at different times are represented as kernel integral equations whose right-hand side, and often the kernel itself, are known approximately from experimental data. The hazard rate function may be estimated by means of the approximate solution of the integral equations. A specific regularization technique for the solution of these stochastic integral equations, which constitutes an ill-posed problem, is required here.

7.2

Statistical regularization method

We begin with the state of the theoretical background of the statistical regularization method which will be required later. Let U and V be metric spaces with metrics

U and V , and A be a continuous one-to-one operator from U to V . We seek the solution g of the operator equation Ag = y

g ∈ U

y ∈ V

(7.2)

for the case where one knows the operators An and functions yn , n = 1 2    , instead of the precise data A y. An and yn are defined on a probability space   P and are close to A and y in some probabilistic sense; here, yn ∈ V , and the operator An is continuous for any  ∈ . The solution gn = A−1 n yn cannot be used as an approximation of g since it is unstable with respect to the variations in the empirical data. More precisely, small deviations of yn from y may lead to large deviations of gn , that is to say, the inverse operator A−1 n may be not continuous. Such a problem is ill-posed (see Definition 13, p. 67). Example 10 Consider the estimation of the PDF fx as a solution of Fredholm’s equation (2.8). For a continuous DF Fx we evidently have fx = F ′ x. If the unknown Fx is replaced by the stepwise empirical DF Fn x the derivative Fn x′ does not exist at a finite number of points corresponding to the jumps in Fn x. Example 11 (Tikhonov and Arsenin, 1977). Let y be an n × 1 vector. If A is an n × n symmetric matrix and det A = 0 (or rank A = n) then A−1 exists. By an orthogonal transformation g = Vg ∗ , y = Vy∗ one can represent A in diagonal form 1      n , where i , i = 1     n, are eigenvalues of the matrix A. Then a linear system Ag = y is represented as i gi∗ = yi∗ , i = 1     n. If rank A = r < n, then n − r eigenvalues of A are equal to zero. Let i = 0 for i = 1     r and i = 0 1

For an individual it might be the states of health, disease and death, or for a technical system the states of good and bad work.

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

183

for i = r + 1     n. For given approximations yn and An such that yn − y ≤  and An − A ≤  ( > 0,  > 0), the eigenvalues i , i = r + 1     n may be i may be large for small yi∗ / close to zero for a sufficiently small . Then gi∗ = perturbations of An and yn . This implies that the solution of the system of linear equations Ag = y is unstable.

The regularization method proposed in Tikhonov and Arsenin (1977) involves the stabilization of solutions using the reduction of the set of possible solutions  ⊆ U to a compact set  ∗ due to the following lemma.

Lemma 3 The inverse operator A−1 is continuous on the set N ∗ = A ∗ if the continuous one-to-one operator A is defined on the compact  ∗ ∈  ⊆ U .

This reduction is provided by the stabilizing functional, which is defined on . The regularization method is similar to the Lagrange method in the sense that we want to find a solution gn which minimizes a functional gn  such that An gn − yn  ≤ ,  > 0. To find the solution of (7.2), we extend the method of regularization from a deterministic operator equation to the case of stochastic ill-posed problems. The function that minimizes the functional R yn  g = An g − yn 2V + g

(7.3)

in a set D of functions g ∈ U is taken as an approximate solution of (7.2). Here,  > 0 is the regularization parameter and g is a stabilizing functional that satisfies the standard conditions (see p. 67). Theorems 15 and 16 provide the theoretical background of the statistical regularization method for the case of an accurately given operator A (Vapnik and Stephanyuk, 1979; Vapnik, 1982), and Theorem 17 for the case of an inaccurately given operator A (Stefanyuk, 1986). Theorem 15 If, for each n, a positive  =  n is chosen such that  → 0 as n → , then for any positive  and  there will be a number N = N   such that, for all n > N , the elements gn x that minimize the functional (7.3) satisfy the inequality 

P U gn  g >  ≤ P 2V yn  y >   (7.4) where g is the precise solution of (7.2) with the right-hand side y, and f g = f − g .

For the estimation of a PDF fx by the regularization method (see pp. 67, 68) one can find the conditions on the regularization parameter n which provide consistent estimates. Let y be the DF Fx, yn be the empirical DF Fn x, V = Ca b and

2V Fn x Fx = supx Fn x − Fx. Taking into account the inequality

P sup Fn x − Fx >  ≤ 2 exp −2n2  (7.5) x

184

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

(Prakasa Rao, 1983), we get from (7.4) that P U fn  f  >  ≤ 2 exp −n 

(7.6)

Hence, if n → 0 and nn →  as n →  then sequence fn x converges in the  probability to the PDF fx in metric space U . If n=1 exp −n <  at least for one  > 0 then the sequence converges with probability one by the Borel–Cantelli lemma. Theorem 16 Let U be a Hilbert space, A be a linear operator, and g = g2U . Then, for any , there exists a number n such that, for all n > n, 



P gn − g 2U >  ≤ 2P 2V yn  y > /2 

Theorem 17 Let U and V be normed spaces. For any  > 0 and any constants c1  c2 > 0, there exists a number 0 > 0 such that, for all  ≤ 0 ,     

yn − y V An − A  P   gn − g U >  ≤ P   > c1 + P   > c2  (7.7) √ √   where

An − A = sup g∈D

An g − Ag V  1/2 g

(7.8)

These theorems imply that the minimization of (7.3) is a stable problem, i.e. close functions yn and y (and close operators An and A) correspond to close (in probabilistic sense) regularized solutions gn and g that minimize the functionals R yn  g and R y g, respectively. For Hilbert spaces U and V , the solution of (7.2) with g = g2U has a simple form, gn = I + A∗n An −1 A∗n yn 

(7.9)

where I is a unit operator and A∗n is the adjoint operator of An . The stability of the approximation gn to g is ensured by an appropriate choice of . Various methods for choosing the regularization parameter were developed, for example, in Morozov (1984), Engl and Gfrerer (1988) and Vapnik et al. (1992). The mismatch method determines  from the equality An gn − yn V = n + n g

(7.10)

where n and n g are known estimates of the data error, yn − y V ≤ n, An g − Ag V ≤ n g; see Morozov (1984). The stochastic analog of the mismatch method is the discrepancy method (2.37). If the operator is defined precisely (n g = 0), then the choice of  from  (7.10) provides a rate 1/2

of convergence of the regularized estimate gn to g that is no better than O  ; see Engl and Gfrerer (1988).

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

7.3

185

Numerical solution of ill-posed problems

Many integral equations such as (7.20), (7.21), (7.37), arising particularly in teletraffic and population modeling, are related to the hazard rate. Hazard rate functions can often be formulated in terms of a Volterra integral equation of the first kind  x Kx tgtdt = yx (7.11) 0

or the second kind

gx −



x

Kx tgtdt = yx

0

(7.12)

where g ∈ U , y ∈ V , and Kx t is a real-valued kernel function. Let U and V be Hilbert spaces. We suppose that these equations can be represented by systems of linear equations. For this purpose, we represent the unknown function g t by a linear combination gˆ t =

N 

j j t

(7.13)

j=1

of N known normalized orthogonal functions j t of a basis in U , for example, in L2 . Laguerre or trigonometric polynomials provide examples of such functions. Substituting (7.13) into (7.11) or (7.12), we get, for i = 1     n, N  j=1

and N  j=1

j



Xi 0

K Xi  t j tdt = y Xi 

  j j Xi  −

Xi 0

 K Xi  t j tdt = y Xi  

respectively. Generally, we obtain a system of linear equations An  = Yn 

(7.14)

where the elements of the n × N matrix An are given, for i = 1     n and j = 1     N , by  Xi aij = K Xi  t j tdt 0

and

aij = j Xi  −



Xi 0

K Xi  t j tdt

in the case of the equations (7.11) and (7.12), respectively. Here An and Yn are random, since they are obtained from a sample X n = X1      Xn . Here

186

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION



T Yn = Y 1      Y n is a random n × 1 vector of observations in the sample points X1      Xn in the presence of stochastic errors, and  = 1      N T is an N × 1 vector of unknown parameters we wish to estimate. For equation (7.20) the vector Yn can be defined as the random vector Fn X1      Fn Xn T and the elements of the matrix An have the form  Xi 1 − Fn t j tdt aij = 0

Here, the unknown DF Ft is replaced by its empirical estimate Fn t. The matrix An may possess eigenvalues equal to or close to zero. In the fist case, the system has no solution, whereas in the second case this solution is a very poor approximation to the real . Roughly speaking, the regularization procedure increases the eigenvalues by adding the regularization parameter  and helps to solve ill-posed and ill-conditioned problems. According to Tikhonov’s regularization method (Tikhonov and Arsenin, 1977), one constructs a regularized approximate solution  ∈ U as the global minimum in U of the smoothing functional R Yn   = An  − Yn 2V +   2U

(7.15)

for a given value  > 0 of the regularization parameter, where  2U satisfies all conditions of the stabilization functional (see p. 67).2 By the common theory proved in Vapnik (1982) for an accurately given matrix A if the function Kx t is precisely known and in Stefanyuk (1986) for an inaccurately given matrix An , the convergence of the regularized estimates to the exact functions in the metric space U is satisfied if  → 0 for increasing sample size n →  (see Theorems 15–17). For Hilbert spaces U and V the minimum of R YN   is achieved (analogously to (7.9) when An is a matrix) by

−1  =  An  Yn  = I + ATn An ATn Yn  (7.16) where I is the identity matrix. The parameters N and  may be selected from a sample from the minimum of the criterion (Michalski, 1987)   Yn − C Yn 2 l2 IN =  (7.17) 1 − n2 trC 

which is a variant of the cross-validation method (Golub et al., 1979). Here C =

−1  An I + ATn An ATn  trC  = N − Ni=1 1 + i /−1 is the trace of C and i are the eigenvalues of the matrix ATn An . The minimization is performed in the region 0 < 2 trC  < n, where the denominator of the expression (7.17) is positive. 2

 2ℓ2 =

N

2 i=1 i

is an example of  2U .

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

187

Finally, the regularized nonparametric estimate is obtained from N 

g  t =

j=1

where  estimates .

7.4

j j t

Estimation of the hazard rate function of heavy-tailed distributions

Without loss of generality we shall now restrict ourselves to nonnegative r.v.s X. For the estimation of the hazard rate hx in the case of heavy-tailed distributions, it is natural to transform an underlying r.v. X into a new r.v. Y = TX with known behavior of hx. It is convenient to consider the transformations T  0  → 0 1 to a finite interval. The transformations considered in Sections 4.2 and 4.3 may be used as Tx. By (3.26) and (7.1) we have hx =

gTxT ′ x = hg TxT ′ x 1 − GTx

(7.18)

x where Gx = 0 gudu, and gx and hg x are the PDF and hazard rate of the transformed r.v. Y . The latter formula is a full analog of (3.26) for PDFs. Like the PDF, the function hx is invariant with respect to monotone transformation in the metric of the space L1 , that is,  1   ˆ hˆ g x − hg xdx hx − hxdx = 0

0

ˆ Here, hx is the hazard rate of the r.v. X, hx is the estimate of hx hˆ g x = g ˆ ˆ gˆ x/1 − Gx is the estimate of h x gˆ x and Gx are the estimates of gx and Gx on [0,1]. This invariance is not valid in the spaces L2 and C. In L2 we get  1   2 ˆ hx − hx dx = hˆ g u − hg u2 T ′ T −1 udu 0

0

≤c if, for any u ∈ 0 1,



0

1

hˆ g u − hg u2 du

0 < T ′ T −1 u ≤ c Obviously,   0

ˆ hx − hx

2

dx ≤ sup hˆ g x − hg x x∈01

(7.19) 

1 0

T ′ T −1 udu

188

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

This implies that the accuracy of the estimation of the hazard rate function on a finite interval determines the accuracy of the estimation on 0 . Evidently, the transformations 2/ arctan x and (4.11) obey the property (7.19). We note that the estimation of the hazard rate hg x of the r.v. Y has the same problems as the estimation of hx for distributions defined on [0,1]. That is, hg x →  holds as x → 1. The proof of this property is provided in Stefanyuk (1992). For example, for the Pareto distribution with DF Fx = 1 − 1 + x−1/ , x ≥ 0, and transformation (4.11) we have by (3.25) that

−1 ˆ ˆ hg x = 2 1 − x2+1 + /ˆ 1 − x − 1 − x2+1  This implies that hg x →  as x → 1. For the Weibull-type distribution with DF Fx = 1 − exp−x   > 0 x > 0, we get  −1 1 − x−2ˆ − 1 ˆ hg x =  1 − x−2−1 ∼ 1 − x−1−2ˆ  ˆ

that is, hg x →  as x → 1 for  > 0. In populations of both living individuals and inaminate objects such as automobile motors a common tendency has been discovered: the mortality risk or hazard rate decreases at infinity, which corresponds to heavy-tailed distributions (Yashin et al., 1996). Below, the accuracy of estimates of hx in the metrics of the spaces L2 and C (Section 7.5) and the ratio between hazard rates of two populations (Section 7.6) are considered in the finite case (for compactly supported distributions), when PDF fx is equal to zero outside a bounded interval.

7.5 7.5.1

Hazard rate estimation for compactly supported distributions Estimation of the hazard rate from the simplest equations

Let us assume that there exists a sample X n = X1      Xn  of independent observations of a r.v. (say, the lifetime of an individual) that takes values in a limited interval 0 d and is distributed with PDF fx and DF Fx, where Fx = 1 for x ∈ 0 d. By definition, the hazard rate hx obeys (7.1). The estimation of this function is hindered primarily by the fact that hx tends to infinity as x → d. Let us represent hx as the solution of the equations  t h x 1 − Fx dx = Ft (7.20) 0

or



t 0

h x dx = − ln 1 − Ft 

(7.21)

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

189

Let us represent these equations in the operator form Ah = y

h ∈ U y ∈ V

where U and V are normed spaces. The form of the operator A depends on the form of the kernel function of the integral equations. To solve (7.20) and (7.21) approximately, the unknown DF Fx is replaced by its empirical estimate Fn x constructed from the sample X n . This means that in the case of (7.20) both the right-hand side and the operator are defined imprecisely, and in the case of (7.21) only the right-hand side is defined imprecisely. Solution of equation (7.20) The estimates of the hazard rate arising from (7.20) were proposed in Stefanyuk (1992). The minimum of the functional 2   d    d − x2 h2 xdx R Fn  h = hx 1 − Fx dx − dFn x +  i

i

i

0

with respect to ht for a fixed  = n > 0 was proposed as an estimate of ht. Here, i are disjoint subintervals covering the interval 0 d. Since hx →  as x → d the weight d − x2 is implemented such that the integral exists. The minimum of the latter functional is reached on the function 1 − Fx  ∗ ci 1 x ∈ i   h∗ x = d − x2 i where

ci∗



dFn x  =   i 2 1−Fx dx +  i d−x

Since Fx is unknown, one may replace it by the empirical DF Fn x or by its piecewise linear smoothing (a polygon). Solution of equation (7.21) The theory for solving (7.21) was developed in Markovich (1998). Note that Fx can be replaced on 0 d by a close function – for example, by the empirical DF Fn x or by a polygon. However, if Fn x is used, then the right-hand side of yn x = − ln1 − Fn x may be unlimited on 0 d, provided that the sample occupies an interval smaller than 0 d. The polygon is close to Fx on 0 d in a linear sense. Suppose it is known in advance that the required function ht is defined on 0 xa , where 0 ≤ xa < d, and Fxa  = a 0 ≤ a < 1. Let a be known from the conditions of the problem, in population analysis for example, 0 < a < 1

190

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

is a proportion of the individuals that died before the age of xa . Then one can replace − ln1 − Ft on t ∈ 0 xa  by − ln1 − Fn t + 1 − t · 1Fn t = 1, where the function t determines the interval of the line connecting the points Xn  Fn Xn−1  and xa  a, X1 ≤    ≤ Xn−1 ≤ Xn are the order statistics of the sample X n , and 1A is the indicator function of the event A, that is, x =

1/n + a − 1 x − xa  + a xa − Xn

holds as x ∈ Xn  xa . Obviously, x ≤ max1 − 1/n a holds, since Xn  = 1 − 1/n and xa  = a. Let ∗ t = 1 − t · 1Fn t = 1

(7.22)

We now give an example of the regularized estimate of the hazard rate that converges to ht in the uniform metric. Let ht satisfy the Hölder (or Lipschitz) condition with the  0 <  ≤ 1, sup h t1  − h t2 /t1 − t2   t1  t2 ∈ 0 xa  t1 = t2 <  which means it belongs to the Hölder space H  0 xa  with the norm   h t1  − h t2   sup h H = sup h x + t1 − t2  x∈0xa  t1 t2 ∈0xa t1 =t2

(7.23)

and let V be the space C0 xa  of functions that are continuous on 0 xa  and have the norm y C = sup yx x∈0xa 

The functional  h = h 2H  which satisfies all necessary conditions (see p. 67) is taken as the stabilizing functional. The regularized estimate h x may be determined by minimizing the functional  2 R yn  h =

sup yn x − yx

x∈0xa 

+  h 

where yx = − ln1 − Fx, yn x = − ln1 − Fn x + ∗ x. By Theorem 15, the estimate h x converges to the required function hx in the metric H  0 xa  and, consequently, in the metric C0 xa . The following theorem, proved in Appendix E, concerns the uniform convergence of the regularized estimates h x to the solution hx of (7.21). Let U = V = C0 xa .

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

191

Theorem 18 If x ∈ 0 xa , where Fxa  = a, 0 ≤ a < 1, h x is the regularized estimate of the function hx, and the regularization parameter  obeys  = n−→0 n→

then





P   lim sup h x − hx = 0 = 1 

n→0x  a

The following lemma, also proved in Appendix E, is required to prove the theorem. Lemma 4 If x ∈ 0 xa , where Fxa  = a 0 ≤ a < 1, then   Fn x − ∗ x − Fx = n y − yn C0xa  ≤ − ln 1 − sup 1 − Fx xFx≤a

The uncertainty of the function y(x) can be estimated by the Rényi statistic (Bolshev and Smirnov, 1965): Fn x − Fx  1 − Fx Fx≤a

Rn 0 a = sup

The value corresponding to the maximum of the PDF of the statistic Rn 0 a can be taken into account in the estimate  of the inaccuracy of n. According to Bolshev na Rn 0 1 − a 0 < a ≤ 1, that corresponds and Smirnov (1965), the value of 1−a to the greatest value of the PDF of the distribution is 09.3 Then, for a∗ = 1 − a, we have    a∗  (7.24)  n = − ln 1 − 09 n1 − a∗  We now turn to the problem of the optimality of the regularization method when we solve (7.21). Let U = V = L2 0 xa , and let hx be approximated by h x A yn , that is, the global minimum of the functional R yn  h = Ah − yn 2 +  h 2 in U for a given value  > 0 of the regularization parameter. Here and henceforth, x · is the norm of the space L2 0 xa . Here, Aht ≡ 0 a t − hd. Since U and V are Hilbert spaces the regularized solution is h A yn  = I + A∗ A−1 A∗ yn  3

The limiting distribution of the Rényi statistic for 0 < a ≤ 1 is   na lim P Rn 0 1 − a < x = 2x − 1 n→ 1−a

where x is the DF of the standard normal distribution (Rényi, 1953).

x > 0

192

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

where A∗ is the adjoint operator of A. Let us characterize the accuracy of the regularization method with a fixed choice of the parameter  = n (n is the sample size) by   = hx − h x A yn   note that the operator B = A∗ A is self-adjoint (Hermitian) with kernel K s = We xa  t −   t − s dt =xa − max s, 0 ≤  s ≤ xa . We denote by 0 < 21 < 0 2 2 <    the characteristic numbers of the positive symmetric kernels that are defined by the operators AA∗ and A∗ A. By k x k = 1 2    and k x k = 1 2    we denote the corresponding systems of eigenfunctions x that are orthonormalized in L2 0 xa . Let i x = i 0 a 1 x ≤ t i tdt. The eigenvalues of the Hermitian operator B are found from sk =   B  = provided that    =

 xa 0

choose systems of functions 

k x

=



xa xa 0

k xKx ! k !dxd!

0

 !2 d! = 1 (Kirillov and Gvishiani, 1982). We

  kx 2 cos  xa xa



k x =



  kx 2 sin  xa xa

k = 1 2    

that are orthonormalized in L2 0 xa . Then xa xa      x 2 kx 1 k! 2  a xa − max x ! cos cos  dxd! = sk = 2 = xa xa xa k k 0 0 (7.25)

Let us assume that the kth k ≥ 1 derivative of hx exists and has a limited variation on 0 xa . Then function hx can be extended to −xa  0 by means of a polynomial rx of order 2k − 1, which is defined by the conditions r0 = 0 r ′ 0 = 0     r k−1 0 = 0 and r−xa  = hxa  r ′ −xa  = h′ xa      r k−1 −xa  = hk−1 xa . The polynomial rx can be extended further to the whole real axis. The set of functions meeting this conditions will be denoted by ℘k . Let us now consider a Fourier series. We note that any yx

∈ L2 0 xa  can be represented by a Fourier series in the orthonormalized basis j in L2 0 xa : yx =

  i=1

ci i x

(7.26)

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

193

The function hx ∈ ℘k is represented by a series in the orthonormalized basis k x k = 1 2    : hx =

 

ai i x

(7.27)

i=1

Under the above conditions where yx and hx belong to L2 0 xa , the series (7.26) and (7.27) converge in the metric of L2 0 xa  and ai = i ci . Since hx ∈ ℘k , the inequality ai  ≤

Vk  xa ik+1

i = 1 2    

(7.28)

where Vk is the variation of hk x, is valid for its Fourier coefficients (Fikhtengol’ts, 1965). Theorem 19 Let X n = X1      Xn  be a sample of i.i.d. r.v.s with PDF fx and DF Fx that are concentrated on 0 d. Let x ∈ 0 xa  Fxa  = a 0 ≤ a < 1 hx ∈ ℘k and the characteristic numbers of the operators AA∗ and A∗ A satisfy (7.25). If, in the regularized estimate h x A yn  of the solution of (7.21),  = n− holds, where 4#/2k + 1 ≤  < 1 − 2# 0 < # < k + 1/2/2k + 3 as k ∈ 0 1

and # ≤  < 1 − 2# 0 < # < 1/3 as k ≥ 2, then the asymptotic rate of convergence of the estimate h x A yn  to hx obeys the expression P   lim n# h x A yn  − hx ≤ c = 1 n→

Here c is constant that is independent on n, and · is a norm in L2 0 xa .

Theorem 19 is proved in Appendix E. It follows from the findings of Ivanov et al. (1978) that there exists no hazard rate estimate with bounded variation of the kth derivative that converges in L2 with a rate better than  nk+05/k+15 .

Remark 11 Let us select n from (7.24), i.e., n ∼ n−1/2 . Then one may observe that the rate of convergence declared in Theorem 19 is optimal in the class ℘k , that is, n−k+05/2k+15 for k = 0 (that is the case for the function hx with a limited variation) and for k = 1.

7.5.2

Estimation of the hazard rate from a special kernel equation

We consider the following problem from population analysis. The cause-specific mortality rate among sick people is the object of the interest. The dynamic of the specific disease (and generally any stochastic changes) in the population may be described by compartmental semi-Markov models of different complexity (or number of states). These models allow us to estimate the cause-specific mortality rate among sick people from mortality and incidence data obtained in the whole

194

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

population or in groups of interest. Similarly, we can estimate the destruction rate from a specific cause among technical systems in a ‘degradated operating system’ state. The semi-Markov property of the models is important because it reflects the natural changes in the properties of the objects due to the fact of being for a particular time in the specific states. Such methodology is important for the investigation of the influence of risk factors such as radiation on the health of a population (Markovich, 1995). Similarly, one can estimate different indices in populations such as the rate of revealed morbidity (Markovich and Michalski, 1995). The two-state model (Figure 7.2) requires the precise monitoring of the lifetimes of sick people, which is expensive. Furthermore, there is a latent morbidity in the population which is not available. Hence, one can consider a three-state model, where there are two life states (‘healthy but at risk’ and ‘sick’) and a death state (Figure 7.3, based on Figure 2 in Markovich et al., 1996). A risk group includes individuals who are in the life states. A transition to the ‘sick’ state is made from the ‘healthy but at risk’ state with rate t. A transition to the ‘death’ state is made from ‘healthy but at risk’ and ‘sick’ states with mortality rate 1 t from causes other than the specific disease – from the ‘sick’ state with the cause-specific mortality rate t. t We denote by St = exp− 0 u + 1 u du the survivor function corresponding to the total mortality rate of the risk group, that is, the probability of a member of the risk group aged x being alive. We denote by P1 t the probability of t being in the ‘healthy but at risk’ state. We denote by gt = t exp− 0 d the PDF of the disease duration for sick individuals before death. The relation between the cause-specific mortality in the risk group t and cause-specific mortality among sick people t is given by Fredholm’s integral equation     x  y  x udu  (7.29) 1 udu g x − y dy = x exp − yP1 y exp 0

0

0

We suppose that the incidence rate in the risk group It is available. Note that yP1 y = IySy Risk group

healthy but at risk

λ(t)

sick

μ1(t) μ1(t)

δ(t) dead

γ (t)

Figure 7.3 Three-state model of survival.

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

on substituting the latter expression in (7.29) we have    x  x udu  Kyg x − y dy = x exp − 0

0

where

Ky = IySy exp



y 0

195

(7.30)

    y udu  1 udu = Iy exp − 0

One can estimate t, using the solution gx of (7.30), by z =

gz z  1 − 0 gydy

An equation similar to (7.30) can be obtained in a more complicated model, where there is additionally a ‘healthy and not at risk’ state (Figure 7.4, based on Figure 3 in Markovich et al., 1996). The difference is that t is the cause-specific mortality for the whole population and rt is the rate of transition from ‘healthy and not at risk’ to ‘healthy but at risk’. The three-state model requires the incidence rate in the special contingents with regard to some risk factors, which is expensive and usually limited in scale. The four-state model uses mortality and incidence data for the whole population from official statistics. We now aim to prove the uniform convergence of regularized solutions of equations such as (7.30). Let X n1 = X1      Xn1  be a sample of i.i.d. observations of an r.v. X that takes values in a bounded interval 0 d and is distributed with continuous PDF yx and DF Hx. For example, this r.v. could be the time to the death after onset of the cause-specific disease in the risk group (three-state model) or in the whole population (four-state model). Let Y n2 = Y1      Yn2  be a sample of i.i.d. observations of the second r.v. that assumes values in the bounded interval 0 d and has continuous PDF fy, the DF Fy, and the hazard rate Iy. Note that Iy = fy/ 1 − Fy. An example Life Risk group

r(t) healthy and not at risk

λ(t) healthy but at risk

sick μ1(t) δ (t)

μ1(t) dead

γ (t)

Figure 7.4 Four-state model of survival.

196

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

of this r.v. is the time to onset of the disease among people at risk (or in the whole population). Let us assume that Hx = 1 and Fx = 1 for x ∈ 0 d. Let Z be an unobservable r.v. that assumes values in the bounded interval 0 d and has continuous PDF gz and hazard rate z. The latter function is to be estimated. The time to death after onset of the cause-specific disease among sick people is an example of the r.v. Z. Let us consider an integral equation x 0

Kygx − ydy = yx

x ∈ 0 d

(7.31)

where Ky = Iy1 − Hy, relative to the PDF of the time of death gz = z hz exp− 0 hd and find hz from the equation hz =

1−

gz z  gydy 0

Let the PDFs yx, fx and gx belong to the space C0 d. We assume that the right-hand side and the kernel of equation (7.31) are unknown and are estimated from the empirical data. For example, some nonparametric estimate (histogram, kernel estimate or, generally, a regularized estimate) yn1 x is given instead of yx. The kernel Ky is also replaced by its estimate, for example, ˆ Ky = In2 y1 − Hn1 y =

fn2 y 1 − Fn2 y + ∗ y

1 − Hn1 y

where ∗ t is determined by (7.22), fn2 x is a nonparametric estimate of fx from the sample Y n2 , and Hn1 x and Fn2 x are the empirical DFs constructed from samples X n1 and Y n2 . The uniform convergence of the regularized estimates g  x to gx in 0 xa , for example, 0 ≤ xa < d and Fxa  = a 0 ≤ a < 1, is the object of interest.

Theorem 20 Let supx∈0xa  fx ≤  for a fixed  > 0 g  x be a regularized estimate of gx obtained by the regularization method, where the stabilizing functional g satisfies the condition min = inf g∈D g > 0,4 and the regularization parameter  = n be n−→0 n→

 

n=1

exp−nn < 

(7.32)

4 One can take the square of the norm (7.23) as g satisfying this condition. Note that g = g 2C does not satisfy the third condition of the stabilizing functional (page 67). In fact, the sequence gn t = c sin nt belongs to the sphere supt gn t ≤ c but does not contain all its limit points, i.e. the set Mc = gn  gn  ≤ c is not compact in C (Vapnik and Stefanyuk, 1979).

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

197

at least for one  > 0, where n = min n1  n2 . Let yn1 x and fn2 x be some estimates of yx and fx, respectively, obtained by the regularization method. Then, P   lim g  x − gx C0xa  ≤ c = 1

(7.33)

n→

where 0 < c <  is a constant.

to prove the theorem. Let Ag =  x The following lemma is required x I y1 − Hn1 ygx − ydy. We denote Iy1 − Hygx − ydy A g = n n 0 0 2 inf 1 − Fn2 x + ∗ x = min 1 − Fn2 xa  1 − a 1/n = C ∗ 

x∈0xa 

(7.34)

Lemma 5 If x ∈ 0 xa , where Fxa  = a 0 ≤ a < 1, and fn2 x is an estimate of the PDF fx, then   An g − Ag C = sup An g − Ag ≤ 2 sup In2 x − Ix x∈0xa 

+

x∈0xa 

  supx∈0xa  fx supx Hn1 x − Hx 1−a



where         f x − fx 1 + Fn2 x − Fx + fx Fx − Fn2 x + ∗ x In x − Ix ≤ n2  2 C ∗ 1 − a Let Gx =

x 0

gd

G x =

x 0

g  d.

Theorem 21 Let the assumptions of Theorem 20 hold, g  x be an estimate of the PDF gx x ∈ 0 xa  such that G x ≤ C < 1 supx∈0xa  gx ≤  for a fixed  > 0 and Gxa  = b 0 ≤ b < 1. Then P   lim h x − hx C0xa  ≤ c = 1 n→

where 0 < c <  is a constant.

7.6

Estimation of the ratio of hazard rates

We consider two r.v.s X, Y (e.g., these might be the survival times in two populations of objects) distributed with PDFs fx, gx and DFs Fx, Gx, respectively. Let X n = X1      Xn , Y n = Y1      Yn  be samples of independent

198

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

observations of these r.v.s on an interval 0 d (in population analysis, d is a maximal survival time); n is the sample size. Generally, the sample sizes of X and Y may be different. Let us suppose that the distributions are compactly supported, that is, fx = gx = 0 if x  0 d. Let us assume that gx = 0 x ∈ 0 d and Fx = 1, Gx = 1 hold for x ∈ 0 d. The hazard rate is defined by 1 x =

fx 1 − Fx

2 x =

gx 1 − Gx

for the first cohort, and by

for the second cohort. In some applications such as hormesis (Markovich, 2000) or failure time detection (Stefanyuk, 1986), it is useful to consider the ratio of the hazard rates r x =

1 x 2 x

(7.35)

or the ratio between two PDFs, the so-called likelihood ratio (see Section 7.6.1): qx =

fx  gx

The problem of estimation results from the fact that rx can tend to infinity as x → d. We can consider the function rx as a solution of Volterra’s integral equation x 0

ru

1 − Fu dGu = Fx 1 − Gu

(7.36)

for x ∈ 0 d. The estimation of rx is an ill-posed problem. In (7.36) the DFs Fx and Gx are unknown. However, they can be replaced by close approximations, for example, by the empirical DFs Fn x =

n 1  x − Xi  n i=1

Gn y =

n 1  y − Yi  n i=1

constructed by the samples X n and Y n . According to the Glivenko–Cantelli theorem the empirical DF is a good approximation of the corresponding DF for sufficiently large n with probability close to one. n u One can take Kn x u = 1−G1−F gn u as a kernel function K x u = ∗ n u+ u 1−Fu gu in (7.36), where g u is some estimate of gx, and ∗ u is determined n 1−Gu

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

199

by (7.22). The term ∗ u is used to prevent a zero denominator of the Kn x u, since Gn x may be equal to one at a b, when the sample Y n occupies an interval less than 0 d.

7.6.1

Failure time detection

Let zt , t = 1 2    , be an observed stochastic process arising from i.i.d. r.v.s with PDF fx. We suppose that at time  a PDF fx changes to gx and fx = gx. We need to estimate this moment , called the failure time (Figure 7.5). Taking into account the independence of zt , the likelihood function of  is given by    T −1   gzt   fzt  ℓ z1      zT  = t=1

t=

Taking logarithms, we get

ln ℓ z1      zT  =

−1  t=1

ln



 T  fzt  + ln gzt  gzt  t=1

The maximum likelihood estimate of  is a value that provides the maximum of the first term on the right-hand side of the latter equation:   −1  fzt  ∗   S = ln → max   gzt  t=1 To find ∗ we first have to estimate qx = fx/gx from the equation  y qxdGx = Fy y ∈ a b

(7.37)

a

10

Z(t)

5

0

–5

0

20

40

60

80

100

t

Figure 7.5 Example of a process with a failure time: fx ∈ N0 1 (solid line), gx ∈ N3 1 (dotted line),  = 50.

200

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

assuming fx and gx are defined at the finite interval a b, gx = 0, x ∈ a b. The DFs Fx and Gx are not known exactly and can be replaced by their empirical estimates. The solution of qx from (7.37) is an ill-posed problem and can be solved by the statistical regularization method (Stefanyuk, 1986).

7.6.2

Hormesis detection

The term ‘hormesis’ is accepted in the modern literature to mean the positive effect on living organisms and constitutions of small doses of toxic substances or stressors, which in larger doses may be unhealthy and destructive. For example, animals preexposed to low-dose radiation may have higher resistance to lethal or sublethal doses (Luckey, 1980). It is well known that high radiation doses may cause illness, but this relationship is not clear for low doses. Of special interest are late effects such as cancer, genetic changes, and congenital malformations. The idea of radiation hormesis arose from the necessity of a natural radiation background for the normal development of organisms, which is ascertained by Planel et al. (1967) from his experiments on fruit flies (Drosophila). This means that if the dose is zero then the mortality rate is not zero. It is incorrect to extend the case of the mortality risk at large doses to small doses. These are commonly considered to be possible hormetic outcomes of low-level radiation exposure: increased longevity, growth and fertility, a reduction in cancer frequency (Sagan, 1987). The term hormesis may well be applied to any physiological effect which occurs at low doses. Studies of the longevity and tolerance of large populations of fruit flies after brief exposure to some heat treatment have shown that during the stress time interval the mortality rate of flies in the stress group was the same as in the control group without the stress treatment. Then, after stress, the mortality rate in the stress group was less than in the control group (Khazaeli et al., 1997). Such an effect may not be explained simply by a selection process, that is, by the presence of the heterogeneity in cohorts. Here, we focus on the problem of hormesis detection in relation to longevity in the case of the possible presence of selection. The lifetimes in both the stress (experimental) group affected by different doses of some stress factor and the control group experiencing standard living conditions (no stress) are observed. To detect the presence of hormesis the ratio between mortalities in the stress and control groups may be useful. Hitherto, most analysts have used the so-called parametric approach to the analysis of stress data. The approach considered in Sachs et al. (1990) includes Markov cell survival models. Continuous-time Markov chain models provide a powerful formulation for incorporating stochastic effects (such as stochastic fluctuations in the number of cell lesions added by an event of given specific energy) into time-dependent radiation cell survival. To detect hormesis according

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

201

to the model, (Yakovlev et al., 1993) the mortality rate is estimated on lifetimes by modeling repair and misrepair processes at the cell level. The capacity of the repair system is estimated as a parameter of the model. It is assumed that the capacity of the repair system of the organism depends on the dose rate. A stepwise reduction of the repair intensity at some nonrandom time instant is suggested. The model is demonstrated on real data, where the decline of the mortality rate for low doses is observed. In the frailty model (Yashin et al., 1996), based on the model of proportional hazards (Cox and Oakes, 1984), a heterogeneity (frailty) stochastic variable is introduced to explain the decline in the mortality rate for low doses of the stress as different reactions of individuals to the same exposure. The problem is that hormesis as well as selection effects may lead to a decline in the mortality rate. The question is how to detect the presence of the hormesis. Yakovlev et al. (1993) indicate the values of parameters showing a strong hormesis effect. The obvious disadvantage of a parametric approach is that the reliability of the results depends on the correspondence of the model to the empirical data. Another problem is that simple models usually do not fit empirical data well, and when one tries to estimate the parameters for more complex models, in most cases the estimates have large variances because of the lack of data. Sometimes it is more realistic to admit some theoretical properties of an unknown function than to define its parametric form. To detect hormesis in the data, we examine nonparametric estimates of the ratio between the mortality in the group under the stress and the mortality in the control group without the stress, which depends on the applied dose of stress, (Markovich, 2000). This ratio may be considered as a dose–effect relationship. Formally, the estimates of this function are obtained as solutions of operator equations. A specific regularization technique for the solution of these equations, which are ill-posed, is described. The estimates are considered for a homogeneous and a heterogeneous population. Models are illustrated on simulated data. Evaluation of the mortalities ratio function for homogeneous populations Suppose that the individuals in the stress and control cohorts are homogeneous in the sense that they have similar reactions on the same stress factor. Let us assume that we observe two r.v.s X and Y . These are individual lifetimes in the stress and control cohorts with PDFs fx and gx and DFs Fx and Gx, respectively, X n = X1      Xn , Y n = Y1      Yn  are samples of independent observations of these r.v.s on the closed interval 0 d, and n is a sample size. Let us suppose that the distributions are finite or compactly supported. This means that fx = gx = 0, when x  0 d. Suppose, that gx = 0 x ∈ 0 d. Furthermore, suppose that Fx = 1 and Gx = 1 for x ∈ 0 d. As in Section 7.6, p. 198, we can consider the mortality ratio function rx (7.35) as a solution of the integral equation (7.36).

202

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

Evaluation of the mortalities ratio function for heterogeneous populations It is sometimes assumed that all individuals in the population are heterogeneous, that is, every individual in a population is supposed to have an individual frailty. Several recent studies show that the presence of heterogeneity in the mortality should be taken into account in order to get a better fit to mortality data (Vaupel et al., 1979). Then the function of the ratio of conditional mortality risks xz for those individuals who have the frailty z is given by r xz =

1 xz  2 xz

(7.38)

Here, we assume that the frailties in both cohorts are identical. Let us suppose that the conditional PDF gxz = 0 for x ∈ 0 d and Fxz = 1, Gxz = 1 for x ∈ 0 d. Let us treat the conditional PDF f xz as a solution of the equation  x  0

0

f uz zdzdu = Fx

(7.39)

where z is some bounded PDF of the frailty z x ∈ 0 d, (for the degenerate distribution, we are led to the case of the homogeneous population), and Fx is unknown. We assume that z is known (e.g., it is a gamma PDF). We can replace Fx by the empirical DF Fn x and determine f xz. After the estimating f xz and g xz from the samples X n and Y n , respectively, one can evaluate the mortality risks for both cohorts using the formulas 1 xz =

1−

fxz x  fuzdu 0

2 xz =

1−

gxz x  guzdu 0

(7.40)

and then determine the ratio of conditional risks from formula (7.38). Sometimes it is more convenient to estimate the function of the conditional likelihood ratio qxz =

f xz  g xz

Numerical solution The integral equations (7.36) and (7.39) can be represented in the operator form (7.2) for the corresponded functions g ∈ U and y ∈ V (Section 7.2). Let U and V be real Hilbert spaces. Then the operator equation (7.2) can be presented as a system of the linear equations (7.14). This system is unstable in view of the inaccuracy of the assignment of empirical data, and so its solution is an ill-posed problem. The solution must be stabilized. The regularized solution  can be found as a global minimum in U of the regularizing functional (7.15) for a fixed value  > 0 of the regularization parameter. For the Hilbert spaces U and V the estimate of the vector

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

203

 is determined by formula (7.16). We will define the matrix An for (7.36) and (7.39). Let us consider (7.36). We will represent the unknown function rt as a linear combination of N known functions, rt =

N 

k

(7.41)

k t

k=1

where k t k = 1     N are (e.g., trigonometric or Laguerre) polynomials, and k  k = 1     N are unknown parameters. Substituting (7.41) into (7.36), we can obtain the elements of the matrix An from the samples X n and Y n : anik

=

Xi 0

1 − Fn u 1 − Gn u + ∗ u

k ugn udu

i = 1     n k = 1     N

where gn x is an estimate of the PDF gx. We define Yn as an n × 1 random vector Fn X1      Fn Xn T . Turning to (7.39), by analogy with the proportional hazards model of Cox and Oakes (1984), we represent the unknown PDF ftz as a linear combination ftz = z

N 

k

k t

(7.42)

k=1

Substituting (7.42) into (7.39), we can obtain elements of the matrix An from the sample X n : aik =

Xi 0

k u



zzdzdu

0

In this case, too, Yn = Fn X1      Fn Xn T . From (7.40) and (7.42) we have 1 xz =

 z Nk=1 k k x x  N 1 − z k=1 k 0 k u du

2 xz =

 z Nk=1 #k k x x   1 − z Nk=1 #k 0 k u du

In a similar manner,

where #k  k = 1   N are built up from the sample Y n . Then,

 x  1 − z Nk=1 #k 0 k u du Nk=1 k k x

 x r xz =   1 − z Nk=1 k 0 k u du Nk=1 #k k x

(7.43)

204

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

Application The estimate of the mortality ratio function is applied to detect hormesis in the possible presence of selection, which is caused by heterogeneity in the population. Two examples illustrate the application of the regularization method to simulated data. One of these is the estimation of the mortality ratio rx = s x/c x, based on (7.36). Here, s x and c x are mortalities observed in the stress and control groups, respectively. The other is the estimation of the conditional mortality in the stress group s xz for a fixed frailty z, based on (7.39) and (7.40). Using estimates of conditional mortalities in the stress and control groups s xz and c xz, one can obtain the conditional ratio rxz = s xz/c xz. All estimates for the stress group are provided for some fixed stress dose rate m. The dose may be interpreted as an exposure. In both examples the unknown functions rx and s xz were obtained as linear combinations of Laguerre polynomials (see (7.41), (7.43)) and the unknown coefficients of the expansions were estimated by means of the regularization method. The regularization parameter was selected by the discrepancy method (2.42), based on the Kolmogorov statistic Dn . A question arises: How useful is the mortality ratio function in helping to detect the hormesis effect caused by a certain agent that is small in some sense? Having a set of samples of individual longevities in the stress group Xmn for different dose rates m = m1      mp and a sample Y n for the control group, one can obtain rx as a solution of (7.36) for every fixed dose. As a result we find a dose–effect dependence rx m. In accordance with the amount of available information, one can operate with the conditional mortality risk ratio rxz derived from (7.38)–(7.40) for various doses and obtain the conditional dose–effect dependence rx mz. The likelihood ratios qx and qxz can also be used to detect the hormesis. Figure 7.6 illustrates roughly the behavior of the mortalities in the stress and control groups actually observed at given stress doses (Khazaeli et al., 1997; Yashin et al., 1996). Let the stress age interval be x1  x2  = 5 10. To obtain these dependencies some functions of the dose rate and age were used. We are interested in relatively small stress doses when the debilitation effect is not observed, that is, the mortality risk does not increase, (Yashin et al., 1996). One should distinguish the behaviors on the stress interval x1  x2  and after this interval, (Yakovlev et al., 1993; Sachs et al., 1990). In view of the interrelation between the biochemical mechanisms of hormesis and selection (Feinendegen et al., 1988), it seems more reasonable to try to establish exact boundaries of the dose interval beyond which hormesis cannot be observed. It is more realistic to point out an approximate dose interval that corresponds to the presence of hormesis and also to determine ‘how far’ the observable process is from ‘pure’ hormesis (without the presence of selection). Figure 7.7 plots the ratio rx m of mortality risks during and after stress. Obviously, if the dose m = 0, then the mortalities in the stress and control groups are the same and r x m = 1. The next dose interval 0 mph = 0 1 corresponds to ‘pure hormesis’ without selection. This means that during a stress

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

205

0.02

0.02 PDF

Mortality risk

0.03

0.01

0.01

0

0

10

20

30

40

0 0

50

5

Age

10

15

Age

Figure 7.6 Model mortality risks (left) and requisite PDFs (right) in the stress group and control group at various stresses: control group (solid line with crosses); stress group at doses 0.5 (dotted line), 2.5 (dashed line), and 4.2 (solid line). Reprinted from Automation and Remote Control, 61(1), Part 2, pp. 133–143, Detection of hormesis by empirical data as an ill-posed problem, Markovich NM, Figure 1. © 2000. With permission from Pleiades Publishing Inc.

1

r(x,m)

r(x,m)

4

2

0.5

0

0

2

4 m

0

2

4 m

Figure 7.7 Model ratio rx m = s x m/c x m against stress dose m: (left) during stress at fixed age x = 7; (right) after stress at fixed age x = 12. Reprinted from Automation and Remote Control, 61(1), Part 2, pp. 133–143, Detection of hormesis by empirical data as an ill-posed problem, Markovich NM, Figure 2. © 2000. With permission from Pleiades Publishing Inc.

time interval the mortality in the stress group will be the same as that in the control group and r x m = 1, but after the stress, the mortality risk in the stress group will be less than that in the control group, and so r x m < 1 (see the curve in Figure 7.6 that corresponds to the dose 0.5). The dose interval mph  msh = 1 4 corresponds to the case where selection and hormesis are present simultaneously (see the curve in Figure 7.6 that corresponds to the dose 2.5). The last dose interval,

206

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

msh  ms  = 4 5, corresponds to the increasing selection process and the lack of hormesis (see the curve in Figure 7.6 that corresponds to the dose 4.2). In this case more frail individuals die in the stress interval and r x m > 1 for x ∈ x1  x2 . As a result, the mortality risk of the surviving individuals in the stress group decreases, but still remains higher than that in the case of the presence of hormesis and selection simultaneously, and then rx m increases for x > x2 . Thus, rx m increases during and after the stress in view of the absence of advantages gained from hormesis. Using rx m, we can determine the dose bounds of the presence of hormesis. If r x m ≈ 1 during the stress and r x m < 1 after the stress, then we may conclude that the empirical data contain pure hormesis. If r x m > 1 during the stress and rx m decreases after the stress (r x m < 1, then hormesis and selection take place simultaneously. If r x m > 1 during the stress and rx m grows after the stress for some m > msh , then hormesis is absent and only selection occurs. A similar analysis can be carried out using the likelihood ratio function. Furthermore, estimates of rx and s xz on simulated data can be obtained. To demonstrate this we generated the samples X n and Y n of lifetimes in stress and control groups, respectively, with sample size n = 10 000, corresponding to the mortality rates in Figure 7.6. The number of basic functions of the approximation (7.41) is N = 8. Figure 7.8 illustrates the simulated mortality risk s xz and ratio rx, corresponding to the stress group given dose m = 25; as well as the corresponding mortalities and ratio, obtained on generated data using the Kaplan–Meier estimator (Cox and Oakes, 1984) and regularized estimates of rx and s xz, obtained for two frailties. To increase the accuracy, the estimates during the stress interval and after this interval are obtained separately.

0.03

3

0.02

Ratio

Mortality

5 1.0

0.01 0

2 1 0

0

2.0 Age

4.0

0

2.0

4.0

6.0

Age

Figure 7.8 Left: Estimates of s x/z for frailties z ∈ 125 15 (dotted line and dashed line, respectively), obtained from formula (7.43); model (solid line) and generated mortality risk s x in the stress group at stress m = 25. Right: Estimated (solid line), generated (dotted line), and model ratio rx = s x/c x of mortality risks for the stress group at stress m = 25. Reprinted from Automation and Remote Control, 61(1), Part 2, pp. 133–143, Detection of hormesis by empirical data as an ill-posed problem, Markovich NM, Figures 3 and 4. © 2000. With permission from Pleiades Publishing Inc.

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

7.7 7.7.1

207

Hazard rate estimation in teletraffic theory Teletraffic processes at the packet level

Figure 7.9 shows the packet dynamics in a packet-switched communication network with routers as switching entitities. Here the packet arrival process at a buffer in front of a transmission link in a router that is modeled as server can most simply be described by a Poisson process. Its intensity  t varies randomly over time, or can be taken to be constant if the arrival process does not vary strongly. It can be modeled as Markov-modulated Poisson process if the intensity is a stepwise random function, or as a more general semi-Markov process. N is the size of the data buffer. The buffer is described as ‘free’ if there are spare places in the buffer. If the buffer is full a new arriving packet is rejected with intensity  t, otherwise the packet is accepted and placed in the buffer. The server picks up packets from the buffer at rate  t and then clears the buffer for new packets. Our problem is to estimate the parameters of packet dynamics from empirical data. The following sources of information may be available: • the time intervals between consecutive arrivals of packets, called ‘interarrivals’ (Figure 7.10); • the number of packets arriving in intervals of fixed time length; • on- and off-periods of traffic generation (Figure 7.10); • durations of full and free buffer regimes; • the number of lost packets in intervals of fixed time length. Here and in Chapter 8 we use empirical data to estimate the parameters of an arrival process (the intensity of a nonhomogeneous Poisson process, the renewal function of a renewal arrival process, and the heavy-tailed distribution of interarrivals between packets or of on-and off-periods), the capacity of a buffer (the buffer overload rate, the mean duration of a full buffer), the loss process (the risk

N λ(t) packets

η(t) Data buffer

μ(t)

Server

Departing packets

Lost packets

Figure 7.9 Integrated system of packet dynamics.

208

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION off-periods

on-periods

t1

t2

t3

t4 Data buffer

Inter-arrivals

Figure 7.10 On- and off-periods and inter-arrival times between packets.

of packet losses, the mean duration of operation without losses) and the capacity of the server.

7.7.2

Estimation of the intensity of a nonhomogeneous Poisson process

First we consider a nonhomogeneous Poisson process N t  t ≥ 0 with intensity function t, t ≥ 0, as a model of an arrival process of events, such as calls in a BISDN or packets in an IP network. The stochastic process N t counts the random number of arrivals before time t provided that the procedure was started at t = 0 with N0 = 0. Since the arrival stream is nonstationary, the probability pi s t = P N t + s − N t = i

of obtaining i arrivals depends on the length s of the observation interval as well as on its starting point t. It is defined by the Poisson distribution pi s t =

$s ti exp −$s t i!

for any i ≥ 0, where $ s t = E Nt + s − Nt =

 

ipi s t

i=1

is the mean number of arrivals occurring in an observation period of length s starting at t. By Gnedenko and Kowalenko (1971, p. 85) we have  t+s  u du = $s t t

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

209

By definition, the intensity function t is equal to the mean number of arrivals per time unit in an infinitesimally small interval t t + s s ց 0, if the derivative of $ s t exists. In practice t often changes periodically or may even strongly increase. However, one just counts the number of arrivals in fixed time intervals. In this case we are interested in estimating t within these intervals using the available empirical data. For this purpose, we subsequently discuss different estimation procedures. First, let us note that the parametric estimates could be useful here to forecast the behavior of t. To estimate the parameters the maximum likelihood method may be applied. However, since the parametric form of t is usually not available, one has to use a nonparametric approach. Let i = t + i − 1  t + i  i = 1     m, denote disjoint subintervals of equal length covering the finite observation interval t t + s. Körner and Nyberg (1993) estimated the arrival rate from the data of one experiment by ˆ = t

m  ki I t ∈ i   i i=1

(7.44)

where ki is the number of arrivals in an interval i and I t ∈ i  is the indicator function of the ith subinterval i . However, the error of this estimate may be very large, since the estimate is constructed by only one random experiment. Let us consider l independent observations of the process in time intervals of fixed length s with the same starting point t, on which the behavior of the observed process is similar. For example, due to the periodicity of the call or session arrival processes in a BISDN or IP network, one may count the number of arrivals over a period of l days within the same fixed time interval t t + s. Let Xi , i = 1     m, denote the generic discrete r.v. counting the observed  number of

arrivals in each interval i , Xi = 0 1 2    . Let Xil = Xi1      Xil be the sample of independent observations of this r.v. and l be the sample size. The ML estimate is given by ˆ = t

m  Xi i=1

i

I t ∈ i  

(7.45)

where the mean values Xi =

l 1 X l k=1 ik

are obtained by the observations Xik  k = 1     l within the interval of the length s = i , (Markovitch and Krieger, 1999). The latter estimate is unbiased, consistent and asymptotically normal. The accuracy of the estimate depends strongly on the choice of the subintervals i .

210

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

7.8

Semi-Markov modeling in teletraffic engineering

7.8.1

The Gilbert–Elliott model

Let us describe the packet arrival and buffer overload process by a two-state Gilbert–Elliott (GE) semi-Markov model (Figure 7.11), where the states are denoted by G and B, and g t and b t are respectively the rates of transition from state G to state B and from state B to state G. The states G and B correspond to ‘good’ and ‘bad’ regimes with a low and high error probability, respectively. That is to say, G denotes a buffer with free spaces and B denotes a full buffer with no free spaces for new incoming packets. If the buffer is full then all new packets are lost and we will consider them as system errors. Consequently, g t is the intensity with which the buffer is filled and reflects the stream of requests, and b t the intensity with which the buffer content is cleared or restored and reflects the capacity of the server. Other interpretations of the GE model are possible. We may consider the generation of packets by a source in a high-speed network. We suppose that G corresponds to the on-period and B to the off-period. Consequently, g t and b t reflect the rates of switching between these two regimes. Usually, the transition rates between states in a GE model are assumed to be constant (Ohta and Kitani, 1990; Bratt, 1994). In our study, however, we consider an inhomogeneous Markov model, where the transition rates depend on time. The model has a semi-Markov property since these rates depend on the time spent in states G and B. For example, the longer the buffer is free, the greater the probability of filling it. Let us describe different approaches to identifying a two-state semi-Markov model from empirical data. Suppose that we observe two r.v.s TG and TB . Commonly these are the durations of the ‘good’ and ‘bad’ regimes of the model – say, of a transmission channel – or the times of the transitions from G to B or from B to G. The r.v.s have PDFs fG x and fB x and DFs P TG < x

and P TB < x , respectively. TGn = TG1  TG2      TGn and TBn = TB1  TB2      TBn are samples of independent observations of these r.v.s. The probability of transition from G to B in the time interval   + d for small values of d is equal to g  PG  d. Then the transition probability in the time interval 0 x is x 0

g  PG  d = PGB x  g (t ) G

B b (t )

Figure 7.11 Gilbert–Elliott semi-Markov model.

(7.46)

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

211

where PGB x = P TG < x and TG is the time of the transition from G to B or the length of time spent in state G. Similarly, the probability of the transition from B to G in the time interval 0 x is x 0

b  PB  d = PBG x 

(7.47)

where PBG x = P TB < x and TB is the time of the transition from B to G or the duration of being in state B. Note that PG x = P TG ≥ x = 1 − PGB x, PB x = P TB ≥ x = 1 − PBG x are the probabilities of being in the ‘good’ and ‘bad’ states, respectively. Let us treat g t and b t as solutions of Volterra’s integral equations (7.46) and (7.47). The problem is that the DFs PGB x and PBG x as well as the probabilities PG x and PB x are unknown. But there are samples of independent observations of the r.v.s TGn and TBn . Therefore, one may replace PGB x and PBG x by the empirical DFs n x = PGB

n

1  x − TGi  n i=1

n x = PBG

n

1  x − TBi n i=1

constructed from the samples TGn and TBn . By the Glivenko–Cantelli theorem the empirical DF converges to the original DF with probability one:   n x → 0 = 1 P sup PGB x − PGB x

n→

x

n→

  n P sup PBG x − PBG x → 0 = 1

This means that for some fixed n the right-hand sides and kernel functions of (7.46) and (7.47) are not precisely known. Thus, one can only obtain approximate solutions gˆ t and bˆ t rather than the exact functions g t and b t. Since g t and b t are hazard rates, we can treat the solution of (7.46) and (7.47) as a solution of integral equation (7.20) and apply the regularization method to estimate the hazard rate (see p. 189). Then one can estimate the PDF of the transition time to the ‘bad’ state from the formula n x fˆGB x = gˆ x 1 − PGB

and the PDF of the transition time to the ‘good’ state from n ˆ x  1 − PBG fˆBG x = bx

The estimation of PDFs fGB x and fBG x may also be treated as the solution of Fredholm’s equation (2.8), using the data TGn and TBn .

212

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

The mean time spent in the ‘good’ state – for example, error-free operation – in the time interval 0 d is given by TG =

d

PG x dx

0

The mean time spent in the ‘bad’ state – characterized by packet loss – is determined by TB =

d

PB x dx

0

The r.v.s TG and TB may be heavy-tailed distributed. On/off-periods are an example of such r.v.s. To estimate the transition rates gt and bt in this case one can transform the data to a compact interval (Section 7.4).

7.8.2

Estimation of a retrial process

Retrial queues are characterized by the following feature: if an arriving call finds the server free, it immediately occupies the server and leaves the system after service completion. But if the server is busy then this blocked call becomes a potential repeated call. The customer may repeat his request after some random delay. This situation plays a special role in several computer and communication networks. Many papers have been devoted to retrial queues (an extensive survey can be found in Falin, 1990), but relatively little is known about the statistics of this problem. Falin (1995) gives an estimate of the retrial rate and its asymptotic variance when the observation period is long. The difficulty of estimating the retrial rate is due to the lack of available empirical information: • In most cases we cannot distinguish primary and repeated calls. • Retrial queues cannot be fully observed (in particular, it is difficult in practice to observe customers in orbit, that is, abandoned customers who are making another attempt to get service). We always observe • a joint arrival flow of primary and repeated calls to the servers, i.e. the number of incoming calls (attempts) in a unit period of time; • holding times of calls (a mean per unit time); • the number of rejected calls (a mean per unit time). Additionally, more detailed information might be measured:

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

213

• time spent in orbit, • times between two consecutive attempts to get service or occupy the same line after the first abandonment (inter-attempts). The latter information could be available from interviews. The time spent in orbit is the time interval between the time of the first case abandonment and the time of the last abandonment (the last unsuccessful attempt to get service) or of the first successful call of the customer. If inter-attempts of several customers are available one can calculate the retrial rate rt directly by one of the methods for the estimation of the hazard rate; see, for example, Section 7.5.1 or Kooperberg et al. (1994). Normally, such information is not available. Due to the lack of information, we elaborate on a probabilistic approach based on the description of a call process at the individual level which progresses in a random jump-like manner and possesses the Markov property. A class of compartmental semi-Markov models is considered for the modeling of stochastic changes of customer states. The main feature of this class is that models of different complexity can be generated from the simpler models by the addition of new states. The semiMarkov property of the models is important since it reflects the dependence of the transition rates between different states of the sojourn times in the states. For example, the retrial rate depends on the length of time spent in orbit: the longer one is waiting for service, the more impatient one becomes. The identification of semi-Markov models uses estimates of transition rates between different states. The problem of model identification is considered in terms of the solution of an integral equation system on the basis of empirical data. The analytical expressions for the probabilities of being in different states are presented in the form of integral relationships. The structure of the model depends on the available empirical data. To construct semi-Markov models it is important to be clear as to what a retrial call is. One may define a retrial call as the attempt of some customer to occupy a fixed channel (or to get service) within a limited time period without attempting to occupy any other channel (definition A). But within the waiting time, the customer may occupy another channel and then return to orbit, that is, restart the attempt to occupy the previous channel (definition B). These different understandings of the retrial call lead to different estimates of the retrial rate.

First Semi-Markov model First, let us accept definition A and formulate the simplest compartment model of the retrial process (see Figure 7.12). This model has three states: ‘server’, ‘orbit’, and ‘out’. Calls in the server state are receiving service. Calls in the out state have either been serviced or are waiting to make their first attempt to get service. The orbit state contains abandoned customers. The rate of the transition from the server

214

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION λ 1(t )

Server

μ(t )

Out

μ0(t )

r (t )

h (t ) Orbit

Figure 7.12 First semi-Markov retrial model.

state to the out state is the service rate t. Abandoned customers are characterized by the rejection rate 0 t, that is, the transition rate from the server state to the orbit state. The arrival rate of primary calls is given by 1 t and corresponds to the transition rate from the out state to the server state. The transition rate rt from the orbit state to the server state is the intensity of repeated calls. The total arrival rate t is given by (7.48)

t = 1 t + rt

The customer may leave orbit, that is, pull out of the next attempt to get service and return to the out state at rate ht – this function thus characterizes the impatience of customers and the abandonment of retrial attempts. Let the functions P1 t, P2 t and P3 t stand for the probabilities of observing a customer at time instant t in the server, out, and orbit states, respectively. Note that P1 t + P2 t + P3 t = 1

(7.49)

Let us consider the structures of these probabilities in more detail. We start with P3 t. Let ut z denote the PDF relating to a customer in the orbit state at time t with waiting duration from z to z + dz. Considering the transitions from server to orbit, from orbit to server, and from orbit to out, we can write ut z = P1 t − z0 t − zSr t zSh t z where

  Sr t z = exp −

t t−z

 rudu 

  Sh t z = exp −

t

hudu t−z



are the probabilities that a customer does not pass to the server and out states in the time interval t − z t.

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

215

The probability P3 t of finding a customer in orbit at time t is equal to the integral of the function ut t − y with respect to y. Here y is the time instant when the transition occurs from the server state to the orbit state:  t P3 t = 0 yP1 ySr t t − ySh t t − ydy (7.50) 0

The probabilities P1 t and P2 t satisfy the differential equations P˙1 t = − t + 0 t P1 t + P2 t1 t + P3 trt

P˙2 t = −1 tP2 t + P1 tt + P3 tht

(7.51) (7.52)

Equations (7.49)–(7.52) form a system that describes the dynamics of the changes in the probabilities of sojourn in different states. The transition rates t, t might be estimated using empirical data, namely from holding times and the number of incoming calls, respectively. The rates ht and 0 t may be estimated qualitatively as a proportion of leaving or rejected customers (e.g., 20% and 3%) or from the lengths of time spent in orbit and the number of rejected calls, respectively (if the latter data are available). Then by (7.48)–(7.52) one can estimate the unknown intensities 1 t and rt, and thus separate the primary and repeated call rates. In the simplest case, one may assume that the holding times are exponentially distributed, and the total arrival stream is a nonhomogeneous Poisson stream. Then t = 1/Th for any t, where Th is the mean holding time, and one can estimate the intensity of the nonhomogeneous Poisson process t by formula (7.45). One can solve equations (7.51) and (7.52) using the following initial conditions: P1 0 = 0 Hence P1 t = P2 t =





t 0

   t 1 yP2 y + ryP3 y exp − u + 0 u du dy y

t 0

P2 0 = 1

   t yP1 y + hyP3 y exp − 1 udu dy y

   t + exp − 1 udu  0

Equation (7.50) describes the probability that a customer is in the orbit state in time t. The probability P3 t may be estimated as the proportion of customers in orbit. In practice, the number of rejected calls per unit time is available. The probability of the transition from the server state to the orbit state in the interval   + d for small values of d is equal to 0 P1 d. Thus, the probability of the transition to the orbit state or the abandonment of the call in the time interval t0  t is  t P ∗ t0  t = 0 P1 d t0

216

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

If Nt is the total number of customers at time t, then the number of rejected calls in time interval t0  t is determined by mt0  t = Nt P ∗ t0  t To estimate P1 t we can now write the equation mt0  t  t 0 P1 d = Nt t0

and use it instead of (7.50). A preliminary estimate of 0  is required. Note that 0  characterizes the probability of the customer being abandoned in time interval t t + t under the assumption that he was not abandoned before t. For small t, 0 t t ≈

mt t + t  Nt∗

where mt t + t is the number of rejections in t t + t, and Nt∗ is the number of calls arriving up to time t. Remark 12 The length of time spent in orbit may have a heavy-tailed distribution. Hence, to estimate the rate ht one can apply the transformation approach (Section 7.4). The total arrival stream may be a renewal process. In particular, its inter-arrival time distribution may be heavy-tailed. To estimate t it may then be useful to apply the methodology described in Chapter 8. Second semi-Markov model Let us now accept definition B. Figure 7.13 shows the semi-Markov model for the retrial call process in this case. This model has the same states as the first one, but some of the transition rates are in principle different. As earlier, t, 1 t, 0 t, ht correspond to the service rate, the primary call arrival rate, the rejection rate, and customer impatience, respectively. Now, however, one may occupy another line or use another service while waiting in orbit. This means that the transition rate from the orbit state to the server state combines the primary call and repeated call rates. Therefore, one may transit from the orbit state to the server state at rate t, satisfying (7.48). After the successful or not successful call one may return, that is, resume the attempt to occupy the former line or access service. This means that one may move from the out state to the orbit state at rate 1 t. Let us now write the system of the equations describing the dynamics of this setup. Let P1 t, P2 t, P3 t again be the probabilities of being in the server, out, and orbit states. Then the following equations hold: P˙1 t = − t + 0 t P1 t + P2 t1 t + P3 tt

P˙2 t = −1 t + 1 tP2 t + P1 tt + P3 tht P˙3 t = −t + htP3 t + P1 t0 t + P2 t1 t

(7.53)

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

217

λ1(t )

Server

λ(t )

μ(t )

μ0(t )

Out

μ1(t ) h(t )

Orbit

Figure 7.13 Second semi-Markov retrial model.

To identify this semi-Markov model one needs some more detailed empirical information. The transition rate 1 t might be estimated quantitatively as the proportion of customers restarting their attempts to get previous service in the time intervals between the end of the successful or unsuccessful call and the attempt to occupy the previous service, if they are available. The rates t, t, 0 t, ht are calculated as described earlier. The description becomes more complicated than the first model, since 1 t depends on the holding times and is restricted by the potential waiting time of the customer. Equations (7.48), (7.49), (7.53) provide the estimation of the retrial rate rt as well as the rate of primary calls 1 t. Remark 13 The solutions of Volterra’s integral equations, using the first and second Semi-Markov retrial call models, are extremely sensitive to noise in the empirical data. The solutions must be stabilized as discussed in Section 7.3.

7.9

Exercises

1. Rough hazard rate estimation. Generate X n according to Pareto distribution (4.8). Estimate the hazard rate function hx = fx/1 − Fx, where fx is the PDF, and Fx is the DF of the underlying r.v., by following procedures: (a) replacing fx by a histogram and Fx by an empirical DF Fn x of the sample X n for x < Xn (for x ≥ Xn  Fn x = 1 and the estimate ˆ hx of the hazard rate is infinite); (b) by the transformation of X n to Y n , Yi = TXi , i = 1     n. The ˆ ˆ −1/2 . Calculate the transformation is given by T = 1 − 1 + x estimate of the EVI ˆ by Hill’s estimator (1.5). Calculate the hazard rate by formula (7.18), where the hazard rate hg x of a new r.v. Y1 = TX1  is estimated by the method indicated in (a).

218

NONPARAMETRIC ESTIMATION OF THE HAZARD RATE FUNCTION

2. Hazard rate estimation from (7.21) by a regularization method. Generate X n according to Pareto distribution (4.8). Transform X n to Y n , ˆ ˆ −1/2 Yi = TXi , i = 1     n by transformation T = 1 − 1 + x . Calculate the estimate of the EVI ˆ by Hill’s estimator (1.5). Estimate the hazard rate hg x by Y n from the equation  t hg xdx = − ln1 − Gt t ∈ 0 1 0

by the estimate by hˆ g x = √ N a regularization method. Determine  x, where k x = 2 coskx k = 1 2      = k=1 kT k −1 T I + An An An zn , the regularization parameter  ∈ 001 01 05 , zn is the vector − ln1 − Gn Y1      − ln1 − Gn Yn  , Gn x is an n empirical  Yi DF with respect to Y , the elements of matrix An are defined as aij = 0 j xdx i = 1     n j = 1     N . Estimate hx by formula (7.18) and plot it for different values of .

3. Hazard rate estimation from (7.20) by a regularization method. Repeat Exercise 2 but estimate the hazard rate hg x by Y n from the equation  t hg x 1 − Gx dx = Gt t ∈ 0 1 0

by a regularization method. The difference is that zn = Gn Y1      Gn Yn  , and the elements of the matrix An are defined as aij =  Yi j x 1 − Gn x dx 0

4. Estimating the intensity function of a nonhomogeneous Poisson arrival process. Consider m disjoint subintervals i = i − 1 i i = 1     m m = 24 (hours), of equal length = 1 covering the observation interval 0 s, s = 24. For each i generate a Poisson distributed random sample of arrivals Xil = Xi1      Xil , l = 100, with intensity i :     2 i−a2 2 1 exp − exp − i−a 2 2 2%1 2%2 +  i = 1     m i =   2 2%12 2 2%22 with parameters a1 = 10 %1 = 3 a2 = 21 %2 = 1, that is, the intensity function t = m i=1 i It ∈ i  is simulated. Estimate t by formula (7.45) and plot it.

8

Nonparametric estimation of the renewal function

We now consider the estimation of the renewal function (RF) within a finite time interval and for infinite time. A nonparametric histogram-type estimator, its asymptotical properties and smoothing methods are presented. The chapter is organized as follows. Section 8.1 provides motivation for RF estimation, and in Section 8.2 the function is defined and its approximations for large time intervals are presented. In Section 8.3 the histogram-type estimate of the RF is described. Section 8.4 provides some results (Theorems 23–26) on the almost sure uniform convergence of this estimate to the RF. The selection of the parameter k of this estimate by the bootstrap method (Section 8.5) and by a plot (Section 8.6) is described for different time intervals. Section 8.7 contains a simulation study on the accuracy of the proposed estimate for different inter-arrival-time distributions and different values of k selected by the bootstrap and plot methods. A comparison with Frees’ estimate is given. An application to TCP flow inter-arrival time data is presented in Section 8.8. Further discussion is provided in Section 8.9. The proofs of the theorems are presented in Appendix F.

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

220

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

8.1

Traffic modeling by recurrent marked point processes

Let us consider measurements at the burst and/or packet level. Then the generated Internet traffic can be characterized by a marked point process N ≡ i  Zi   i = 1 2     of inter-arrival times  i of bursts (or, in a more general setting, of IP packets) of sizes Zi . Then tn = ni=1 i  t0 = 0, is the nth arrival time based on the sampled inter-arrival (or inter-renewal) times 1      l  with an assumed absolutely continuous DF F (x). Then the overall volume of IP packets or Web traffic in an observation period 0 t is determined by Vt =



iti 0,   Nt  Zi = EZi ENt  = EZHt EVt = E i=1

(Trivedi, 1997). Here Ht = ENt  denotes the RF of the corresponding nonmarked arrival process i  i = 1 2     of transferred files and pages, respectively. The definition of the RF is presented in Section 8.2. For all fixed t > 0 the variance of the overall volume is determined by: varVt = varZi Ht + EZi 2 varNt  if EZi2 <  holds (Trivedi, 1997). It may be evaluated computationally at any t > 0 by the estimation of Ht and the expectation and variance of Zi . Hence, a key issue concerns the estimation of the RF for moderate time intervals 0 t . The approximation of the variance varNt  for large t,  2  2 2 3 5 4 2 3 varNt  ≈ 3 t + + + 4− 3 

2 4 4 3

where , 2 , and 3 are the first three moments of the inter-arrival time distribution, is given by Heyman and Sobel (1982). We also consider an estimate of the RF using a limited number of independent observations of the inter-arrival times 1  2      1  for an unknown interarrival-time distribution (ITD). The nonparametric estimate is derived from the representation of the RF as series of DFs of consecutive arrival times using a finite summation and approximations of the latter by empirical DFs. Due to the limited number of observed inter-arrival times the estimate is accurate only for closed time intervals 0 t . An important aspect is determined by the selection of an optimal number of terms k of the finite sum. Two methods are proposed:

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

221

(i) an a priori choice of k as a function of the sample size l which provides almost surely the uniform convergence of the estimate to the RF for lightand heavy-tailed ITDs if the time interval is not too large, and (ii) a data-dependent selection of k by a bootstrap method and by a plot. To evaluate both the efficiency of the estimate and the selection methods of k, a Monte Carlo study is carried out.

8.2

Introduction to renewal function estimation

Renewal processes have a wide range of applications in warranty control, in the reliability analysis of technical systems and particularly of telecommunication networks such as high-speed packet-switched networks like the Internet. Normally, measurement facilities count the events of interest – the number of requested and transferred Web pages, incoming or outgoing calls, frames, packets or cells in consecutive time intervals of fixed length. It is important for planning and control purposes (e.g., for intrusion detection) to estimate the traffic load in terms of the mean numbers of events counted and their variances in these intervals. In such applications the RF constitutes the basic characteristic of an underlying renewal process since by means of this function the expectation and variance of the number of arrivals of the relevant events before a fixed time instant can be calculated (Gnedenko, 1943; Feller, 1971). To estimate the RF, several realizations of the counting process may be required – for example, the observations of the number of calls over several days. Here we consider the estimation of the RF using inter-arrival times between events for only one realization of the process. Let Ft = P n < t, with F0+ = 0, denote the common DF of the i.i.d. inter-arrival times n  n = 1 2     of these events. The renewal counting process Nt  t ≥ 0 denotes  the number of events before time t Nt = maxn  tn < t for t ≥ 0, where tn = ni=1 i  t0 = 0, are the arrival times. The RF Ht is expressed by Ht = E Nt  =

 

n=1

P tn < t =

 

F ∗n t

(8.1)

n=1

for t ≥ 0, where F ∗n denotes the n-fold recursive Stieltjes convolution of F . Several RF estimation methods have been developed for a known ITD. Unfortunately, explicit forms of the RF are obtained only in rare cases, for example, if the inter-arrival times have a uniform distribution, or for the wide class of matrixexponential distributions (exponential and Erlang distributions belong to this class); see Asmussen (1996). Therefore, several attempts have been made to evaluate the RF computationally; see Chaudhry (1995), Deligönül (1985), McConalogue (1981), and Xie (1989). In many problems, Smith’s (1954) key renewal theorem may be useful.

222

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

Theorem 22 Let the distribution Ft be continuous F0 = 0 F = 1 and Qt ≥ 0 be a monotone nonincreasing integrable function on 0 . Then lim



t

t→ 0

Qt − dH =

1  Qxdx

0

Example 12 One can get Ht = t for the exponential distribution using Qt −  = t − .

We denote the mean inter-arrival time by = En . If the variance 2 = varn  of F is finite, then, applying Smith’s theorem, the RF Ht may be approximated for large t by the expression Ht =

2 1 t + 2 − + o1

2 2

widely used in the literature. If is finite, but 2 is not finite, then Ht ∼

t + GF t 1

t → 

where GF t =

1  t   1 − Fx dx dy

2 0 y

Note that GF t →  as t →  if and only if 2 =  (Sgibnev, 1981). For regularly varying distributions 1 − Fx = x− ℓx and some 1 <  < 2, where ℓx is a slowly varying function, it was shown that Ht ∼

t t2 1 − Ft + 2 

 − 12 − 

t→

(Teugels, 1968). This result has been extended to the case 1 <  ≤ 2,

 y t 1  t 1 − Fx dx dy Ht − ∼ 2 t → 

−

0 0 in Mohan (1976). In Chaudhry (1995) a closed-form   expression for the RF is stated if the Laplace–Stieltjes transform f s = 0 e−st dFt, Res > 0, of Ft is a rational function. It is assumed that f s may be represented by the ratio f s =

1

Ps Qs

The notation ∼ means that the ratio between the two functions of variable t converges to 1 as t → .

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

223

of two polynomials Ps and Qs of degree k and less than k, respectively.2 Denoting by si  i = 1 2     k, those roots of the polynomial Qs − Ps with Resi  ≤ 0,3 we obtain Ht = Ak t + where Ai =

Q′ s

k−1  i=1

k−1  Ai Ai expsi t −  si i=1 si

Psi   ′ i  − P si 

i = 1 2     k

It can be shown that Ak = 1/ . For large t, one can use the asymptotic expression in terms of roots Ht ≃ Ak t −

k−1  i=1

Ai  si

Asymptotic estimates do not perform well for small t relative to , which is especially important for the load control of telecommunication systems and the warranty control of devices (Frees, 1986a). In practice, it is a more realistic for the distribution to be unknown or only general information describing it to be available. The estimation of the DF or the PDF, if the latter exists, may become complicated if the distribution of the r.v. is heavy-tailed (Section 3.1). Here we focus on the estimation of the RF with no information on the form of the underlying distribution and we use only a sample T l = n  n = 1 2     l of the nonnegative i.i.d. inter-arrival times between events of size l. The nonparametric estimate (8.4) is related to a histogramtype estimate where the unknown probabilities P tn < t in (8.1) are replaced by the corresponding empirical DFs and a limited number k of terms are used in the summation. A similar nonparametric estimate, Hl t k =

k 

n

Fl t

(8.2)

n=1

was proposed by Frees (1986a,b) and further investigated in Schneider et al. (1990). These authors used −1  l n Fl t = t − i1 +    + in  n c  as an

estimate of the arrival-time distribution. Here c denotes the sum over all n l n distinct index combinations i1  i2      in  of length n. The U -statistic Fl t 2

Matrix-exponential distributions have rational Laplace transforms, whereas the Weibull distribution does not have a closed-form Laplace–Stieltjes transform. 3 The main problem is to estimate the roots accurately.

224

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

is a minimum-variance unbiased estimator of F ∗n t. In contrast to estimate (8.4), which uses just one combination of adjacent inter-arrival times, the computation of the Hl t k is awkward. The accuracy of such types of estimates depends on k. Frees (1986b) obtained the almost sure uniform consistency of Hl t k on compact intervals 0 t  0 ≤ t < , under the assumptions that k = l and Ft has a positive mean and finite variance, and the asymptotic normality of Hl t k for each fixed point t > 0. Under some moment conditions on the r.v. min0 i , the almost sure consistency and the asymptotic normality are proved for real-valued inter-arrivals i (Frees, 1986b). However, the data-dependent selection of k (which is important for moderate samples) was not considered in Frees (1986a, b, or Schneider et al. (1990). Grübel and Pitts (1993) proved the convergence of a noncomputational empirical RF Hlg t =

 

n=1

Fˆ l∗n t

(8.3)

on R as l → . Here, Fˆ l∗n t is the n-fold convolution of the empirical DF Fl t based on the sample T l . In our discussion of the estimate (8.4) below, an unbiased estimate of F ∗n t is used, but its variance is not minimal. This inaccuracy is compensated by the data-dependent selection of k and the use of larger samples. An a priori choice of k as a function of the sample size l is considered to obtain almost surely the uniform convergence of the estimate to the RF as l →  for light- and heavytailed PDFs. The bootstrap and plot methods are applied for a data-dependent choice of k.

8.3

Histogram-type estimator of the renewal function

We consider the estimate of the RF Ht which was first introduced in Markovitch and Krieger (2002b) and investigated in Markovich (2004) and Markovich and Krieger (2006a, b). Let r denote the integer part of a real number r. Substituting N , t the empirical mean the empirical ln for E i t we replace the DF P n < ti by  DF Fln t = l1 i=1 t − tn  (an unbiased estimate), where tn = n·i q=1+ni−1 q , n i = 1     ln  ln = nl , n = 1     k, are the observations of the r.v. tn . Then we can estimate the renewal function Ht based on the  samples of independent  l l renewal-time observations t1 = t11      t11      tk = tk1      tkk by  t k l = H

ln k  1 t − tni  l n n=1 i=1

(8.4)

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

225

 t k l = k holds for t ∈ tmax k , where tmax k = Note that H max1≤n≤k max1≤i≤ln tni and k is some fixed number. Errors arise in the estimation from both the approximation of Ht in (8.4) by a finite sum and the approximation of P tn < t by the empirical DF Fln t:      k       (8.5) P tn < t − Fln t   P tn < t + Ht − H t k l  =   n=k+1 n=1  t k l as well as the estimator (8.2) are From this formula one can see that H biased since k is limited. A rough upper bound for the bias is given by  t k l = biast k l = Ht − E H

 

n=k+1

P tn < t ≤

 

n=k+1

Ftn =

Ftk+1  1 − Ft (8.6)

For small t, F(t) is generally small and Ft < 1, thus, this error is small. To provide a good approximation of P tn < t by the empirical DF, according to the Glivenko–Cantelli theorem sufficiently large values ln should be used k < l. Note that lk = 1 for l/2 < k ≤ l, that is, the sample tk contains only one point. Therefore, it is reasonable to take k ≤ l/2. In the following, we provide an optimized estimate of k. On the other hand, to provide a good approximation of Ht by  t k l in general, the value of k should be large enough. Therefore, means of H  t k l is sensitive to the choice of k and the length of the estimation the estimate H  t k l may only be accurate within the interval 0 t . Obviously, the estimate H interval 0 tmax k , since the sample size l is limited.

8.4

Convergence of the histogram-type estimator 4

The convergence of the estimator (8.4) to the RF in the metric of the space C of continuous functions (Theorems 23–25) was proved in Markovich and Krieger (2006a). To estimate the risk (8.5), one is interested in the  t-regions 0 t ⊆ 0 tmax k . The main problem is to estimate the systematic error  n=k+1 P tn < t. To do so, one needs some information about the DF Ft of the r.v. , perhaps the existence of the moment generating function (Cramér’s condition); see Petrov (1975). Then one can use precise large-deviation results for P tn< t. The r.v.  satisfies Cramér’s condition if there exists  > 0 such that E e < . Cramér’s condition is equivalent to an exponential decay rate of 1 − Ft and is satisfied for light-tailed distributions; it provides the existence of all moments of the r.v. .

4 With the exception of Theorem 26 and Corollary 3, this section is taken from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Section 2.1, © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

226

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

Theorem 23 Let 1      l  be a sequence of i.i.d. nonnegative r.v.s and t ∈ 0 tmax k . We suppose that Ei m <  for some integer m ≥ 1, Ei = , vari  = 2 , and that the parameter k obeys k = kl ∼ l as l →  Then

0 <  < 1/3

(8.7)

   t k l  = 0 = 1 P   lim sup Ht − H l→

t

The rate of this uniform convergence may be proved for the class  S of ITDs such that 1 − Ft ≥ exp−t

for any t ∈ 0 T and some  > 0. We assume, without loss of generality, that 0 T = 0 1 . The class  S includes, for example, the exponential distribution and the Weibull distribution with shape parameter greater than 1. Hence, it follows for the estimate of the right-hand side of (8.6) that Ftk+1 1 − exp−tk+1 ≤  1 − Ft exp−t Then, for Ft ∈  S the error of an approximation by (8.4) in the metric of C is estimated by  t k l  sup Ht − H t

(8.8)

  k   1 − exp−tk+1  ≤ sup + P tn < t − Fln t  n=1  exp−t t 

Theorem 24 If 1      l  is an i.i.d. nonnegative sample with DF F ∈  S, t ∈ 0 1 and the parameter k = c · l (c = c > 0), 0 <  < 1/3 − 2/3, 0 <  <  t k l to Ht is 1/2, then the asymptotic rate of convergence of the estimate H given by the expression    t k l  ≤ c1 = 1 P   lim sup l Ht − H l→

t

where c1 is a constant that is independent of l.

Then the following confidence interval is derived for the RF. Corollary 2 If the assumptions of Theorem 24 hold, then, with probability at least 1 − , 0 <  < 1,  t k l − D ≤ Ht ≤ H  t k l + D H

(8.9)

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

where D=l

−



+k −

227

k ln /2  2l

In practice, inter-arrival times are often described by distributions with heavy tails (Chistyakov, 1964; Goldie and Klüppelberg, 1998). Two classes of heavytailed distributions are well known: the distributions with regularly varying tails where 1 − Ft = t− ℓt, t > 0  > 0, and ℓx is a slowly varying function; and the subexponential distributions, with the property that for any  > 0 there exists T = T F such that for any t > T , 1 − Ft > exp−t. It is the specific feature of heavy-tailed distributions that they do not satisfy Cramér’s condition. If t is not too large, an approximation   2 of Ptn > t by the tail of the standard ¯ normal distribution x = √12 x e−y /2 dy is used for heavy-tailed distributions, namely,

t ¯ Ptn > t ∼  √ (8.10) n ¯ √t  − 1 = 0) if t ∈ 0 cn /hn  for (this means that limn→ sup0 2, cn /hn may be ∼ n05 ln05 n, (Mikosch, 1999). Theorem 25 If 1      l  is a sequence of i.i.d. nonnegative r.v.s with heavytailed DF Ft, t ∈ 0 mintmax k ck /hk  , and the parameter k obeys (8.7), then    P   lim sup Ht − H t k l  = 0 = 1 l→

t

The next theorem, proved in Markovich (2004), gives the rate of uniform convergence for distributions with regularly varying tails. By the Karamata representation theorem (Embrechts et al., 1997; Mikosch, 1999; Resnick 2006) the slowly varying function ℓx can be rewritten in the form  x  y ℓx = cx exp dy  x ≥ x0  (8.11) y x0

for some x0 > 0, where c· is a measurable nonnegative function such that limx→ cx = c0 ∈ 0  and x is continuous function, limx→ x = 0. It is assumed that cx is a monotone decreasing or increasing function and x is a nonpositive function. Theorem 26 Let 1      l  be i.i.d. nonnegative regularly varying r.v.s with tail F x = ℓxx− , x > 0,  > 0, parameter k = d · l , d ≥ −A, 0 ≤  < 1 − 2/3

(8.12)

228

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

A = A =

 +  − 1 +  ln l − ln c∗ +  ln x0   + 1 − ln 1 − c∗ l1+−+ x0

c∗ = minc0  ca,  > 1/ − ∗  0 < ∗ < , 0 <  < 1/2,  ∈ x0  tmax k , x0 > 0, t ∈ a tmax k , a > 0. Then     P   lim sup l Ht − H t k l  ≤ c1 = 1 l→

t

where c1 is a constant that is independent of l.

Corollary 3 If assumptions of Theorem 26 hold,  > 1 − ln /ln l/ − ∗ , 0 < ∗ < , then with probability at least 1 − , 0 <  < 1, inequality (8.9) holds, where 

 − l1−−∗  k − D = l + k − ln  2l 2 The theorems determine the values of k as functions of the sample size l. These values k are given only up to a rough asymptotic equivalence. For instance, k can be multiplied by any positive constant and the theorems remain valid. In practice, one needs exact optimal values of k which are adapted to the empirical data. Therefore, we subsequently consider a data-dependent selection of k.

8.5

Selection of k by a bootstrap method5

Using empirical data, k can be chosen automatically by minimizing the bootstrap estimate of the mean squared error of Ht for fixed t (Markovich and Krieger, 2006a),5 i.e.  t k l − Ht2 → min  MSEt k l = EH k

The bootstrap estimate is obtained by drawing resamples with replacement from the original data set T l . Some observations from T l may appear more than once, while others do not appear at all. The bias of the estimate (8.4) is given by (8.6) and the variance by 2   t k l2  − E H  t k l ˜ k l = EH var Ht 2  k k k    Ptn < t  = Pmaxtn  tm  < t − n=1 m=1

5

n=1

This section is based on Markovich and Krieger (2006a, Section 2.2).

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

229

For the solution of several problems of statistics such as the choice of smoothing parameters of the kernel estimator for a PDF, a nonparametric regression or Hill’s estimate of a tail index, it is recommended to use smaller resamples of size l1 < l. The goal is to avoid the situation where the bootstrap estimate of the bias is equal to zero regardless of the nonzero true bias of the estimator (Hall, 1990). The bootstrap estimate of the RF that is constructed from T l by some of the resamples Tl∗1 = 1∗      l∗1  of size l1 is given in a way similar to (8.4) by 1

 ∗ t k1  l1  = H

ln k1  1 t − tn∗i  1 n=1 ln i=1

ln1 = l1 /n 

tn∗i =

The values l1 and l may be related by l1 = l  

ni 

q∗ 

q=1+ni−1

0 <  < 1

(8.13)

The values k1 and k are related by k = k1 l/l1  

0 <  < 1

(8.14)

What values of  and  should be taken? Considering the related problem of choosing the smoothing parameter in the case of a Kernel PDF estimator or linear regression, Hall (1990) has shown by means of asymptotic theory that  = 1/2 leads to the most accurate results. For the bootstrap estimation of the parameter k of Hill’s estimate,  = 2/3 has been recommended.  ∗ t k1  l1  are given by The bias and variance of H and

 ∗ t k1  l1  T l  − H  t k l b∗ t k1  l1  = EH

 2  ∗ t k1  l1 2 T l  − EH  ∗ t k1  l1  T l   var ∗ t k1  l1  = EH

(8.15)

(8.16)

respectively. Here, T l is fixed and the expectation is calculated over all theoretically possible resamples Tl∗1 of T l with size l1 . The bootstrap estimate of MSEt k l is determined by  ∗ t k1  l1  − H  t k l2 T l  MSE∗ t k1  l1  =EH    2  ∗  t k l E H   ∗ t k1  l1 T l =E H t k1  l1  T l − 2H  t k l2  +H

 t k l2 does notdepend on k1 , the problem reduces to the minimization SinceH 2     t k l E H  ∗ t k1  l1 T l with respect to k1 .  ∗ t k1  l1  T l − 2H of E H

230

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

In the following, we will show that minimizing MSE∗ t k1  l1  with respect to k1 is as awkward as the calculation of the estimator (8.2). The problem arises from the calculation of the subsequently considered statistic Fln t. It coincides with the n statistic Fˆ l∗n t – see (8.3) – and it is close to the U -statistic Fl t. We note the difference that Fln t is calculated over all combinations of n observations with possible repetitions. Since ln is the number of possible tn∗i , we get   ln1 ln1 k1 k1    1 1 ∗ l ∗i l  EH t k1  l1  T  = E t − tn T = Et − tn∗i T l  1 1 n=1 ln i=1 n=1 ln i=1 n

k1 k1 l   1 i Fln t t − t  = = n n n=1 n=1 l i=1

(8.17)

Hence, by (8.15) and (8.17) one can see that the bias of the bootstrap approach does not depend on l1 , that is, b∗ t k1  l =

k1 

n=1

 t k l  Fln t − H

(8.18)

When resamples of size l are used, then the bias of the bootstrap b∗ t k l =

k 

n=1

 t k l Fln t − H

may be close to zero for sufficiently large l (since Fln t and Fln t may not differ so much), but for k = 1 it is equal to zero since ln = ln regardless of the true bias  t 1 l (see (8.6)).6 of H Independent of the values k1 and l1 , the bootstrap variance is equal to zero. This property follows from (8.16), (8.17) and the expression  ∗ t k1  l1 2 T l  = EH

1

1

n

m

ln lm k1 k1  1    Et − maxtn∗i  tm∗j T l  1 l1 l n=1 m=1 n m i=1 j=1

k1 k1 k1 k1 l  l    1   i j Fln tFlm t t − maxt  t  = = n m n+m l n=1 m=1 i=1 j=1 n=1 m=1

One should not mix the variance and the bootstrap variance. The latter is a r.v. The bootstrap estimate of MSEt k l is given by MSE∗ t k1  l =

6

k1 k1  

n=1 m=1

See also Example 2 in Section 1.2.2.

 t k l Fln tFlm t − 2H

k1 

n=1

Fln t

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

231

However, the fact that the statistic Fln t cannot be computed easily is a problem. Using the empirical estimate of b∗ t k1  l, B 1  k l bˆ ∗ t k1  l1  = H b t k1  l1  − Ht B b=1

(where B denotes the number of l1 -sized resamples), instead of the actual bootstrap bias may give rough results. Here, we denote by H b t k1  l1  the estimate (8.4) constructed from some resample. In practice one can minimize the empirical estimate of MSE∗ t k1  l, ∗

∗  t k1  l1  =   t k1  l1  MSE b∗ t k1  l1 2 + var

where

(8.19)

2  B B  1  1  t l1  k1  = var H b t k1  l1  − H b t k1  l1  B − 1 b=1 B b=1 ∗

is an empirical estimate of the bootstrap variance. All possible values of k1 should be examined, where k1 is an integer in the interval  1 /2

.  1 l The estimate Hl t k requires A1 = kn=1 nl n + 1 = 2l 1 + l/2 −   l  l l  k l requires A2 = − n=k+1 n nl − 1 operations, whereas Ht n=k+1 n k l  n=1 n n+1 operations. The selection of k in Ht k l by means of an empirical bootstrap method (i.e., the minimization of (8.19)) requires A2 + A3 operations,  l /2  k   where A3 = k11=1 B n11 =1 nl1 n1 + 1 + 6 . Note that 1

Sk =

k  1 1 = c + k +  n k n=1

where c ≈ 05772 is Euler’s constant, z =  ′ z/z, and · is the gamma function (Prudnikov et al., 1981). We suppose for simplicity, that l/n = l/n, l1 /n1 = l1 /n1 , l1 /2 = l1 /2 and k = l/2 = l/2. Since

l/2

 l 1 + −1l l l−1 =2 +  4 l/2 n=0 n

(Prudnikov et al., 1981), we have A1 = 2l−1 1 + l +





l    1 + −1l l l −1− n  A2 = l l/2 + Sl/2  4 l/2 n n=l/2+1   1 /2 l 2 + l1  l Sk1 + 3l1  + A3 = Bl1 1 8 k1 =1

232

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

Hence, for example when k = 10, l = 20 we have A1 = 586 · 106 , A2 = 258579, A1 /A2 = 2266 · 104 . Let B = 50, l1 = 6 ≈ l06 then A3 = 3118 · 103 and A1 /A2 + A3  = 1735 · 103 (see Table 8.1).

8.6

Selection of k by a plot

As a practical tool for the visual selection of k, the plot of the histogram-type estimate for a fixed t against k may be used. An example of a similar approach is given by the Hill’s plot for selecting the tail index. The idea of the plot is based on the uniform convergence of estimate (8.4) to the true RF as k increases and l → . Then one can select for a fixed t the smallest k corresponding to the interval of stability of the plot, namely,  k l = Ht  k + 1 l k = 1     l − 1 k∗ = arg mink  Ht

(8.20)

(Markovich, 2004). Figure 8.1 shows plots of the histogram-type estimate (8.4) against k for different fixed time intervals 0 t , t ∈ 1 3 5 10. The Weibull distribution with shape parameter s = 3 and the sample size l = 50 is considered. The table to the right of the figure shows the values k selected by the bootstrap with parameters  = 05 and  = 07 and by the plot. In this example, the bootstrap recommends larger k than the plot method. The choice of k is determined by the trade-off between two terms in the sum (8.5). The smaller k corresponds to a larger bias and a better estimate of the DF by the empirical DF. A larger k leads to a reduction in the bias. In Figure 8.2 the histogram-type estimates for a Weibull (s = 3) and a gamma (s = 055  = 1) distribution are shown. The parameter k is selected by the

Bootstrap Plot

Hest(k)

10

5

0

0

5

10

t

k

k

1

2

2

3

6

4

5

9

7

10

18

13

15

k

Figure 8.1 Histogram-type estimate of the RF against k for a Weibull distribution: t = 1 (solid horizontal line), t = 3 (dotted line), t = 5 (dot-dashed line), t = 10 (solid line).

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

233

4

1

Hest(t)

Hest(t)

3 0.5

2 1

0 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 t

1

1.1 1.2

0 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 t

1

1.1 1.2

Figure 8.2 Histogram-type estimate of the RF against k for Weibull s = 3 (left) and gamma s = 055  = 1 (right) distributions and the corresponding RFs. k is selected by the bootstrap method (solid line) and by the plot method (dotted line); the sample size is l = 50. The corresponding lines coincide for a Weibull case. The values of RFs (dot-dashed line) were taken from Baxter et al. (1982).

bootstrap and by the plot method. The corresponding curves coincide for Weibull distribution. Using larger sample sizes l and an adaptive selection of k from the data T l , the histogram-type estimate may provide a smaller mean squared error compared to Frees’ estimate (Markovich, 2004). In contrast to (8.4), Frees’ estimate requires a lot of calculations and hence cannot be evaluated for large sample sizes.  k l with The numbers of operations required to calculate Hl t k and Ht fixed, bootstrap-selected and plot-selected k are shown in Table 8.1. The plot method can also be applied to Frees’ estimate. Table 8.1 Number of operations for Frees’ and the histogram-type estimator with fixed, bootstrap-estimated, and plot-estimated k. Estimator

Frees’ estimator with fixed k Histogram-type estimator with fixed k Histogram-type estimator with bootstrap-selected k Histogram-type estimator with plot-selected k

No. of operations

  A1 = kn=1 nl n + 1= l l 2l 1 + l/2 −  l  n=k+1 n − l − n=k+1 n n − 1  A2 = kn=1 nl n + 1

A2 + A3 ,  l1 /2

A k =1  3 = k1 1  l1  B n1 =1 n n1 + 1 + 6 1  l/2

A = 2 k=1  l/2 k l n=1 n n + 1 k=1

Example k = 10, l = 20, B = 50, l1 = 6 586 · 106 258

3376

1544

234

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

8.7

Simulation study 7

To evaluate the performance of the bootstrap approach, we have to investigate the influence of the values  and  in (8.13) and (8.14) by a Monte Carlo simulation. We consider the values  ∈ 01 02 03 04 05 06 07 and  ∈ 03 05 07. Let 0 T be the interval of the estimation, where T ∈ 05 25 45 65 85 10. We generate samples with known PDFs   exp −t t ≥ 0 f1 t = 0 t < 0 of an exponential distribution with parameter  = 1,  ts−1 −s exp−t//s f2 t = 0

t > 0 t ≤ 0

of a gamma distribution with parameters s = 2 and  = 1, and  t>0 sts−1 exp −ts   f3 t = 0 t≤0 of a Weibull distribution with s = 05. The latter distribution is heavy-tailed and subexponential. It is one of the most interesting distributions in reliability engineering where the PDF is singular. For f1 t and f2 t the RFs are determined by H1 t = t

H2 t = 05 t − 05 + 05 exp−2t 

respectively. For f3 t the explicit form of the RF is unknown. Therefore, we use the results of a numerical approximation by Xie’s RS method for H3 t since this method provides rather accurate results for a known PDF and a correctly selected step size h = t/N . Here N is the number of points inside the interval 0 t (Xie, 1989). Strictly speaking, Hi is recursively calculated by  F1 i + i−1 j=1 Fi − j Hj − Hj − 1 − F0 Hi − 1  Hi ≈ 1 − F0   i+05t where 0 = z0 < z1 <    < zN = t and Hi = H i·t  Fi = F  F0 = N N    it  F 05t  F i = F are used. 1 N N Tables 8.2 and 8.3 show the bias and mean squared error of the estimate (8.4) calculated by 200 repeated samples for the given f1 t and f2 t. The parameter k in

7

This section is taken from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Section 3. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

235

Table 8.2 Quality of estimate (8.4): gamma s = 2  = 1 E = 2, sample size l = 50. T

0.5

2.5

4.5

6.5

8.5

10



01 02 03 04 05 06 07 01 02 03 04 05 06 07 01 02 03 04 05 06 07 01 02 03 04 05 06 07 01 02 03 04 05 06 07 01 02 03

 = 03

 = 05

 = 07

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

−2699 26.3 23.3 59.3 −3157 14.3 −687 −280 −530 −110 −140 76.86 64.45 −6884 −1840 −1400 −240 260 120 −261 110 −4700 −3970 −330 −280 −870 410 −7029 −7070 −8920 −1720 −1140 290 −130 150 −11340 −17940 −4560

17.5 20.06 15.67 23.45 19.62 18.14 19.59 210 170 290 250 250 230 260 910 1130 800 670 690 600 630 4360 4770 1540 1600 1260 1340 1270 11100 18220 3530 3570 2220 2150 2430 18530 46720 8850

82.3 −1699 18.3 −267 39.3 50.43 32.3 −270 −270 −390 −120 36.35 83.03 −7626 −1660 −1310 −1010 −200 99.1 230 −1040 −3090 −2680 2330 −320 −390 −540 260 −6160 −6060 −5280 −710 −170 −170 320 −8180 −7290 −10660

21.36 19.09 19.05 18.03 17.09 23.12 20.51 200 230 230 250 270 270 280 750 650 850 610 710 890 670 2560 2800 3310 1410 1330 1500 1510 7920 10400 11640 2480 2730 2040 2360 14530 16290 29870

45.3 −137 29.3 14.3 −237 14.3 −367 −220 −230 −260 −9868 −4005 −320 −6847 −570 −760 −670 24.27 −190 −130 −350 −1300 −1630 −1500 −880 −540 −5784 360 −2870 −2620 −2470 −2540 −840 −1940 280 −5550 −4260 −4670

17.18 16.74 19.21 17.74 18.17 19.82 15.91 260 200 230 250 280 250 270 690 660 620 880 760 780 720 1570 1570 1770 1820 1940 1800 1410 4130 4320 4350 5870 1900 5630 2210 7060 8400 1120

236

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

Table 8.2 (Continued) T

 = 03



04 05 06 07

 = 05

 = 07

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

−4370 −340 87.66 −240

8770 3360 3240 2680

−1120 −1620 300 −570

3290 4510 2750 3200

−4750 −6370 −3920 450

13830 18730 11810 3180

Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Table 1. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

Table 8.3 Quality of estimate (8.4): Exp  = 1 E = 1, sample size l = 50. T

0.5

2.5

4.5

6.5



01 02 03 04 05 06 07 01 02 03 04 05 06 07 01 02 03 04 05 06 07 01 02

 = 03

 = 05

 = 07

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

−170 −250 63.17 −170 38.92 −9496 −1429 −5580 −4290 −810 −1760 −300 −240 200 −13190 −14970 −5920 −6370 −1530 −340 −340 −29770 −43990

120 140 180 130 140 200 160 4620 4430 2240 1990 1500 1780 1900 24050 35240 11690 12122 4914 5970 6050 93881 196426

−280 −120 −110 −180 160 96.71 39.46 −4160 −4750 −4600 −520 −360 −9167 −310 −11850 −12300 −11990 8570 8980 −690 −580 −17910 −25250

130 130 160 180 160 160 160 2920 3780 4100 1705 1800 1780 1790 21340 27260 34250 7344 8064 5530 4760 53084 96100

−220 −100 −220 −87 −4112 −2137 83.21 −2760 −3100 −3300 −2060 −1620 −1480 10.83 −8880 −8190 −11140 15500 −8940 −10160 −840 −16540 −17600

130 140 140 170 150 130 170 2190 2250 2809 2540 2520 2340 1980 12660 14160 23220 24025 23720 28880 4750 45496 56929

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

8.5

10

03 04 05 06 07 01 02 03 04 05 06 07 01 02 03 04 05 06 07

−29560 −30890 −16080 −5660 −970 −47860 −65030 −54830 −54720 −43610 −29300 −15120 −60990 −8000 −70040 −70030 −60020 −48530 −35700

98470 104522 37133 11534 7691 232324 422890 301401 300523 194481 97281 34040 372954 160000 490560 490420 360600 239806 137418

−25900 −10460 −7080 −3200 −1290 −28200 −44620 −51080 −27000 −24240 −12490 −8290 −46240 −65010 −75810 −55120 −52790 −30320 −28160

121243 38887 31541 15951 12454 116417 242950 325356 143110 127377 54102 39720 246810 447293 598302 370272 348808 163944 150466

−20780 −25240 −24060 −24000 −3420 −24310 −29930 −35690 −40360 −4220 −39760 −16810 −32180 −43990 −48620 −51640 −56520 −55660 −17800

237 79919 111020 112225 110622 20050 98658 148687 206116 266565 279417 269464 89880 152412 287296 345162 424061 387048 451449 141676

Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Table 2. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

(8.4) is determined by the bootstrap method. The sample size is l = 50, and B = 50 bootstrap resamples were taken. To understand better the results of Tables 8.2 and 8.3, it may be helpful to examine Figures 8.3 and 8.4 and Table 8.4. The values MSE and BIAS are the averages of MSE and BIAS values over all different T for each fixed couple  . In Figures 8.3 and 8.4 the left-hand figures correspond to Table 8.2 and the right-hand figures to Table 8.3. Table 8.4 shows the corresponding smallest values of MSE and BIAS for a gamma and an exponential distribution. From Tables 8.2–8.4 and Figures 8.3, 8.4 it is evident that •  = 07,  = 03 give the smallest MSE;

• the best trade-off between the averages MSE and BIAS is provided by  ∈ 06 07,  ∈ 03 05;

• the mean squared error increases if the time interval 0 T of the estimation is extended.

Figures 8.5–8.7 show Xie’s estimate and the histogram-type estimate for the PDF f3 t and the PDF ⎧   ⎨ c  c+1  t > 0 f4 t =  +t ⎩0 t ≤ 0

238

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION 20

1.5

Mean MSE

Mean MSE

15 1

0.5

10

5

0

0

0.2

0.4

0.6

0

0.8

0

0.2

0.4

0.6

0.8

alpha

alpha

0.6

3

0.4

2

Mean Bias

Mean Bias

Figure 8.3 Averages of the MSEs from Tables 8.2 and 8.3 over different T for fixed  = 010107,  ∈ 03 05 07 and for a gamma distribution (left) and an exponential distribution (right):  = 03 (solid line),  = 05 (dotted line),  = 07 (dot-dashed line). Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Figure 1. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

0.2

0

0

0.2

0.4 alpha

0.6

0.8

1

0

0

0.2

0.4

0.6

0.8

alpha

Figure 8.4 Averages of the BIASes in Tables 8.2 and 8.3 over different T for fixed  = 010107,  ∈ 03 05 07 and for a gamma distribution (left) and an exponential distribution (right):  = 03 (solid line),  = 05 (dotted line),  = 07 (dot-dashed line). Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Figure 2. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

239

Table 8.4 Minimal values min MSE and min BIAS of averages MSE and BIAS calculated by Tables 8.2 and 8.3 and corresponding  and . Gamma distribution min MSE min BIAS

0.121 7757 · 10−4

  = 07 03   = 06 05

Exponential distribution 3.121 0.643

min MSE min BIAS

  = 07 03   = 07 05

Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Table 3. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

5

4

H(t)

3

2

1

0

0

1

2

3

4

5

t

 k l against t, with Figure 8.5 Estimation of the RF of a Weibull distribution by Ht  = 07,  = 03 (step line);  = 07,  = 05 (dot-dashed line);  = 07,  = 07 (solid line with circles);  = 01,  = 05 (solid line with crosses);  = 04,  = 05 (dotted line). Xie’s estimate is shown by the solid line. Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Figure 3. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

of a Pareto distribution with parameters c = 05,  = 05 on the time interval 0 5 , as well as for the exponential PDF with  = 1. The sample size is l = 100. The parameter k was selected by the bootstrap method with parameters   ∈ 07 03 07 05 07 07 01 05 04 05. For the bootstrap B = 50 resamples was taken. In Figures 8.6 and 8.7 the lines corresponding to   ∈ 07 03 07 05 07 07 coincide with each other. In Figure 8.5

240

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION 1.5

H(t)

1

0.5

0

0

1

2

3

4

5

t

 k l against t, with  = 07, Figure 8.6 Estimation of the RF of a Pareto distribution by Ht  = 03 (step line);  = 07,  = 05 (dot-dashed line);  = 07,  = 07 (solid line with circles);  = 01,  = 05 (solid line with crosses);  = 04,  = 05 (dotted line). Xie’s estimate is shown by the solid line. Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Figure 4. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group. 5 4

H(t)

3

2 1 0

0

1

2

3

4

5

t

 k l against t, with Figure 8.7 Estimation of the RF of an exponential distribution by Ht  = 07,  = 03 (step line);  = 07,  = 05 (dot-dashed line);  = 07,  = 07 (solid line with circles);  = 01,  = 05 (solid line with crosses);  = 04,  = 05 (dotted line). Xie’s estimate is shown by the solid line. Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Figure 5. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

241

the line corresponding to   = 07 07 differs from those lines with   ∈ 07 03 07 05 (the latter two lines coincide) approximately on the interval [2.3,3]. One can see that the curves corresponding to k selected by bootstrap with   ∈ 07 03 07 05 for f3 t and with   ∈ 07 03 07 05 07 07 for the Pareto and exponential PDFs are closer to the true RF than all other curves. The line with   = 04 05 is better than that with   = 01 05, especially for the Pareto PDF. The figures support our previous conclusion regarding the prevalence of  = 07 and  ∈ 03 05. The figures also illustrate the following phenomenon. Referring to formula  t k l may not change (8.4), one can see that for some fixed t the value of H anymore as k increases. For example, we have for f3 t (Figure 8.5) and t = 3 that k ∈ 3 3 26 29 36 for   ∈ 01 03 04 05 07 07 07 03 07 05, respectively. This reflects the situation where the corresponding tni  are larger than t and the corresponding terms in the sum (8.4) are equal to 0. The second part of the simulation study relates to the comparison with the tables presented in Frees (1986a).For this purpose samples of the  lognormal distribution −1 √ with PDF fx = x 2 exp − log x − 2 /2 2  , where = 0 and 2 = 1, and of the Weibull distribution with s = 3 (not heavy-tailed Weibull) were generated. The gamma distribution (s = 055  = 1) presented in Frees (1986a) was not considered. Since the generator of gamma r.v.s (Ahrends and Dieter, 1974) used in Frees (1986a) is not reliable for small samples, it has an adverse influence on the accuracy of the results of a simulation study. T ∈ 025 05 075 10 125 are the times provided and l ∈ 10 15 20 25 30 100 are the sample sizes. As in Frees (1986a), two characteristics, the bias and the mean squared error of the estimates, were calculated over 500 Monte Carlo repetitions and Ht is the true RF. For the fixed number of points T and for the distributions mentioned, Ht was taken from tables in Baxter et al. (1982). The results of the calculation are presented in the Tables 8.5–8.8. Frees’ results are included here; H3n t denotes the estimate (8.2). Since (8.2) requires much computational effort, only k ∈ 5 10 and l ≤ 30 were considered. Considering (8.4), the parameter k is calculated by the bootstrap method, that is, by minimizing (8.19) with respect to k1 , where l1 and k1 are related to l and k by the formulas (8.13) and (8.14) with parameters  = 07 and  = 03. B = 50 bootstrap resamples was taken. The parameter k is also calculated by the plot method. Tables 8.5–8.8 show that, for all estimates: • the mean squared error increases as T increases; • for any fixed T the mean squared error decreases as l increases; • the bias does not exhibit stable behavior.

242

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

Table 8.5 Monte Carlo study of the bias and the mean squared error of Frees’ estimate ˜ k l of the renewal function for the lognormal H3n t and histogram-type estimate Ht distribution with parameters  = 0 = 1 and E = exp1/2 for sample sizes l ∈ 10 15, different time intervals 0 T and values of the parameter k. size

T

 k l Ht

H3n t k=5

k = 10

BIAS ·104

MSE ·104

BIAS ·104

10

025 05 075 1 125

−1 149 128 163 117

77 255 422 659 918

−1 149 128 −163 117

15

025 05 075 1 125

1 123 99 133 93

54 179 296 443 619

1 123 99 123 93

kboot

MSE ·104

BIAS ·104

77 4424 255 5569 422 4335 659 6368 918 −5165

54 1021 179 5968 296 −3354 443 −100 619 3589

MSE ·104

kplot

BIAS ·104

MSE ·104

7725 270 610 800 1220

−7661 12.444 93.918 −165865 100.713

87147 252072 52139 766749 1227

5347 190 360 620 890

−20967 −61409 −14722 −97716 49.006

47905 17174 373013 564595 776039

Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Table 5. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

Table 8.6 Monte Carlo study of the bias and the mean squared error of Frees’ estimate ˜ k l of the renewal function for the lognormal H3n t and histogram-type estimate Ht distribution with parameters  = 0 = 1 and E = exp1/2 for sample sizes l ∈ 20 25 30 100, different time intervals 0 T and values of the parameter k. size

T

 k l Ht

H3n t k=5

kboot

kplot

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

55.214 −115966 −65762 −73716 −72275

44624 127713 236264 387564 590623

20

025 05 075 1 125

–10 94 95 141 126

38 124 218 329 452

−5257 −2215 59.99 −6107 55.8

36.04 130 280 390 600

25

025 05

16 111

29 101

16.62 −3762

29.71 100

−40129 4.626

31635 99883

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION 075 1 125

123 179 185

188 277 384

30

025 05 075 1 125

9 79 67 96 112

23 80 151 218 305

100

025 05 075 1 125

n.a. n.a. n.a. n.a. n.a.

n.a. n.a. n.a. n.a. n.a.

30.21 110 −120

15.63 −3147 −2983 −6682 −100

−4998 −2453 −1675 −2161 −2067

200 340 520 26.05 77.29 150 320 430 7.425 27.44 52.76 88.89 120

243

206168 315193 505936

−73613 −44874 −54876

9.638 −23484 11.417 1.324 101.473

24501 81438 168952 285543 410146

−6862 9.008 −20798 −69252 −76327

8223 2814 58039 8698 115574

Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Table 6. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

Table 8.7 Monte Carlo study of the bias and the mean squared error of Frees’ estimate ˜ k l of the renewal function for the Weibull H3n t and histogram-type estimate Ht distribution with parameter s = 3 and E = 089 for sample sizes l ∈ 10 15, different time intervals 0 T and values of the parameter k. size

T

H3n t k=5

10

025 05 075 1 125

15

025 05 075 1 125

k = 10

kboot

 k l Ht

kplot

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

−12 9 16 −8 −35

15 98 238 318 307

−12 9 16 −8 −35

15 98 238 318 307

−8295 −1933 7.938 16.52 34.9

13.79 100 260 370 440

12.02 110 270 380 420

9 71 177 211 219

−6965 28.48 −4044 9.58 71.88

9.494 79.1 190 260 320

−2826 −4927 −5993 −1445 −1023

−17 −13 36 9 −20

9 71 177 211 219

−17 −13 36 9 −20

15.66 −3178 41.87 76.49 86.14

10.83 73.78 160 250 300

Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Table 7. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

244

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

Table 8.8 Monte Carlo study of the bias and the mean squared error of Frees’ estimate ˜ k l of the renewal function for the Weibull H3n t and histogram-type estimate Ht distribution with parameter s = 3 and E = 089 for sample sizes l ∈ 20 25 30 100, different time intervals 0 T and values of the parameter k. size

T

H3n t k=5

20

25

30

100

025 05 075 1 125 025 05 075 1 125 025 05 075 1 125 025 05 075 1 125

kboot

 k l Ht

kplot

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

BIAS ·104

MSE ·104

−12 1 43 34 35 −13 −12 1 −12 13 −14 −39 −35 −46 −47 n.a. n.a. n.a. n.a. n.a.

7 53 130 174 171 6 38 104 130 125 5 31 90 114 111 n.a. n.a. n.a. n.a. n.a.

−2307 12.61 47.86 −1442 25.26 16.46 13.47 −9219 12.23 −149 −2973 −2265 −2533 −5866 16.28 8.271 6.621 −7625 22.16 −6199

7.818 59.16 130 200 230 6.059 48.87 120 160 200 5.554 38.53 92.53 130 160 1.582 11.29 29.3 46.03 67.3

10.67 −2199 24.9 −2074 −5525 −3505 25.38 −4171 59.9 31.88 −1642 51.86 −2707 63.76 10.29 −1908 19.59 −2898 −8576 −2928

7.862 61.22 130 190 230 5.819 47.21 120 170 210 4.715 32.65 90.16 140 190 1.522 11.6 30.78 45.14 61.91

Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Table 8. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

 k l with H3n t, one may conclude that: Comparing Ht

 k l and H3n t are comparable • the biases and mean squared errors of Ht for the same sample sizes; • the increasing the sample size provides better accuracy with regard to  k l as shown in Table 8.9, where MSEH , MSEH and BIASH , Ht 3n 3n BIASH are averages of the MSE and BIAS over different T . The averaging was provided using the results of Table 8.6 and 8.8 when the sample size is  equal to 30 in the case of H3n and to 100 for H.

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

245

 for the sample Table 8.9 Comparison of H3n t for the sample size 30 and Ht size 100 regarding averages of the MSE and the BIAS for different distributions and two selection methods of k (the bootstrap and the plot method). Distribution

MSEH3n /MSEH

BIASH3n /BIASH

bootstrap

plot

bootstrap

plot

262 226

262 232

270 170

199 204

Lognormal Weibull

Reprinted from Stochastic Models, 22(2), pp. 175–199, Nonparametric estimation of the renewal function by empirical data, Markovich NM and Krieger UR, Table 4. © 2006 Taylor and Francis Group, LLC. With permission from Taylor and Francis Group.

Comparing the bootstrap and the plot methods from Table 8.9, one may conclude that both methods demonstrate similar MSE and BIAS.

8.8

Application to the inter-arrival times of TCP connections

The estimator (8.4) was applied to 1000 TCP flow inter-arrival times measured in a mobile network (Markovich, 2008). Table 8.10 gives the descriptive statistics of this sample. To apply (8.4) we need to check the independence of the inter-arrivals. Tests (see Section 1.3) show that the inter-arrivals of TCP connections are heavy-tailed distributed (Figure 8.8). The distribution of inter-arrivals is close to an exponential one. They may be independent (Figure 8.9). The parameter k in (8.4) was calculated by formula (8.20). The estimate looks close to a straight line on time interval [0.5, 5] but on smaller time intervals the curve is not quite linear (Figure 8.10). This implies that the interarrivals of TCP connections cannot be considered as a pure Poisson process. Formula (8.4) provides a more exact estimate of the mean number of TCP connections.

Table 8.10 Description of the TCP flow inter-arrival times data. Unit

Min.

Max.

Mean

Variance

sec

10−5

2.237

0.235

0.085

246

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION 0.8

0.5 0.6

0.3

EVI

e_n(u)

0.4

0.4

0.2 0.2

0.1 0

0

0.5

1 u

1.5

2

0

0

100

200 k,m

300

400

Figure 8.8 Left: Sample mean excess function en u (1.41) against the threshold u for the inter-arrivals of TCP connections. The plot is close to a constant, which indicates closeness to the exponential distribution. Right: EVI  estimation by Hill’s (dotted line) and group estimator (solid line) for the inter-arrivals of TCP connections. Horizontal lines show the plot-selected values: ˆ H n k = 0388 and ˆ G n k = 0356. The latter values imply infinite first two moments of the inter-arrival distribution and, hence, heaviness of the tail.

0.1

ACF

0.05

0

–0.05

–0.1

0

200

400

600

800

1000

h

Figure 8.9 The sample ACF (1.43) of the inter-arrivals of TCP connections. The horizontal √ dotted lines indicate 95% asymptotic confidence bounds ±196/ n corresponding to the ACF of i.i.d. Gaussian r.v.s. The inter-arrivals are located inside the confidence interval and, hence, may be independent.

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION 30

H(x,a)

4

H(x,a)

247

2

20

10

0

0

0.5 a

1

0

0

2

4

6

a

Figure 8.10 Histogram-type estimate (8.4) of the RF constructed by inter-arrivals of TCP connections for the time interval [0,0.75] (left) and [0,5] (right).

8.9

Conclusions and discussion

In this chapter, we have discussed nonparametric estimation of the RF which does not require any knowledge of the form of the ITD. Due to the limited number of empirical data the histogram-type estimate (as well as Frees’ estimate (8.2)) can be applied for closed time intervals 0 t with relatively n small t. Compared to Frees’ estimate Fl t of the arrival-time distribution F ∗n t in (8.2), the estimate (8.4) uses a simpler and rougher estimate of F ∗n t. The RF Ht is approximated by a finite sum of estimates of the arrival-time distributions with k terms. The parameter k is selected to compensate the error of the risk function. The estimate (8.4) may be computed for sufficiently large l and k, which is not realistic for (8.2). Theorems 23 and 25 state, both for heavy- and light-tailed ITDs, those values of the parameter k as functions of the sample size which provide almost surely the uniform convergence of the histogram-type estimate (8.4) to the true RF for sufficiently small t. It is proved that a smaller value of k k < l than in Frees (1986b) is sufficient to get a reliable estimate of the RF. In Theorem 24 the rate of uniform convergence and a confidence interval of the RF for the specific class of ITDs with an exponential decay rate of the tails are presented. But these theorems determine k only up to a rough asymptotic equivalence. Such a value k does not depend on the empirical data. This feature may influence the accuracy of the estimation. To estimate k from samples of moderate size, the bootstrap method is used. Following Hall (1990), a smaller resample size l1 (and k1 ) is used to avoid the situation where the bootstrap estimate of the bias is equal or close to zero  regardless of the true2bias of the estimate. Then the bootstrap estimate  ∗ t l1  k1  − Ht  l k T l  of the mean squared error MSEt k l = E H  2  l k − Ht  is minimized with respect to k1 , where H  ∗ t l1  k1  is the E Ht

248

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

RF estimate derived from one of the resamples. The relevant relationships between l and l1 as well as between k and k1 are found by a Monte Carlo study. As an alternative data-dependent method to estimate k from samples of a moderate size, the plot method is proposed. The histogram-type estimate (8.4) tends to decrease the MSE in comparison with Frees’ estimate (8.2) by using larger samples. The number of operations required for (8.4) with the bootstrap selection of k is much less than that for Frees’ estimate, and with the plot selection is less than that for a bootstrap selection. As usual, the main disadvantage of the bootstrap method is that it requires the choice of additional parameters  and  that determine the resample size to estimate k. In contrast, the plot method does not require any additional parameters.

8.10

Exercises

1. RF estimation at infinity. Generate 100 inter-arrivals which are regularly varying with DF Fx = 1 − x− 

x ≥ 1

for  = 3 and 1 <  < 2

Approximate the behavior of the RF Ht for large t (e.g., 100 < t < 10 000) applying the formulas Ht ≃ t/ + 2 /2 2  − 1/2   Ht ≃ t/ + t2 / 2  − 12 −  1 − Ft 

for for

 = 3 (8.21)

1 <  < 2 (8.22)

Determine the true values of the mean and variance of Fx. Calculate the empirical mean and variance as estimates of and 2 . Estimate Fx in (8.22) by the empirical DF and by Fˆ x = 1 − x−ˆ . Estimate the tail index  = 1/ by Hill’s estimator, ˆ = 1/  H n k. Compare the estimates and the approximations (the latter are calculated for the known  2 and Fx). 2. RF estimation at infinity. Generate l = 1000 Weibull distributed random numbers with s = 03 (p. 234) as T l . Estimate the tail index by some method and determine the number of finite moments. Approximate the behavior of the RF Ht at infinity by formulas (8.21) and (8.22) if possible. 3. RF estimation on a finite interval 0 t . Generate l = 100 exponentially distributed random numbers with  = 05 and l = 100 gamma distributed random numbers with s = 2 and  = 1. Calculate a histogram-type estimate of the RF on 0 t  t ∈ 05 075 1 125 by formula (8.4). Take different k ≤ l/2. Compare the estimates with the true RFs, Ht = t and Ht = 05 t − 05 + 05 exp−2t, of the exponential and gamma distributions.

NONPARAMETRIC ESTIMATION OF THE RENEWAL FUNCTION

249

4. RF estimation on a finite interval 0 t . Generate random numbers as indicated in Exercise 3. Construct the plot  t k l against k for different 0 t  t ∈ of the histogram-type estimate H 1 2 3 5 10. Determine the value of k by the plot method for each fixed t, that is, by formula (8.20). This implies that for each curve related to a fixed t the value k = kt is selected that corresponds to the begining of a stable interval of the curve.  t k l with time-dependent k. Compare the estimates with Calculate H the true RFs of the exponential and gamma distributions.

Appendix A

Proofs of Chapter 2

Proof of Lemma 2. By (2.48), the left-hand side of (2.49) clearly does not exceed n    Kh x − u Kh x − v fi u v − fufvdudv i=1

≤ n

 

Kh x − u Kh x − v dudv ≤ h + 2 n 

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

Appendix B

Proofs of Chapter 4

Proof of Theorem 3. Suppose that h∗ → 0 as n → . This implies that for any integer N > 0 ∃n > N such that h∗ = h∗ n > H∗ , where H∗ is some positive constant. We shall prove that, for such h∗ , supx∈∗ Fn x − Fh∗ x → 0 as n → . For any solution h∗ and x ∈ ∗ one may represent the divergence in (4.28) using the substitution u = t − Xi  /h∗ :    n  1 1  x t − Xi Fn x − Fh∗ x =  x − Xi  − K 1t − Xi  ≤ h∗ dt n i=1 h∗ 0 h∗   ti2 n  1 Ku1u ≤ 1du   x − Xi  − = n i=1 ti1 where we denote ti1 = ti1 h∗  = −Xi /h∗ , ti2 = ti2 h∗  = x − Xi  /h∗ , and x =



1 0

x ≥ 0 x < 0

Furthermore, we omit 1u ≤ 1, bearing in mind that Ku is compactly supported on −1 1. Without loss of generality, we can consider the sequence h1 ≤ h2 ≤ ≤ hj ≤ 

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

254

PROOFS OF CHAPTER 4

where hj = h∗ nj  = H∗ + j  is some positive constant, and N < n1 ≤ n2 ≤ ≤ nj ≤ . Since, for any fixed i, the sequences −Xi /h∗ nj  → 0 and x − Xi  /h∗ nj  → 0 as nj → , we have  ti2 Kudu → 0 as nj →  ti1

Therefore,

 x − Xi  − Hence,



ti2

ti1

Kudu →  x − Xi  

sup Fn x − Fhj x → 1

x∈∗

nj → 

as nj → 

Therefore, h∗ → 0 as n → , since supx∈∗ Fn x−Fh x → 0 as n →  according to (4.28). Proof of Theorem 4.

Obviously,

sup Fn x − Fh x ≤ sup Fn x − Fx + sup Fx − Fh x

x∈∗

x∈∗

(B.1)

x∈∗

We shall estimate supx∈∗ Fx − Fh x. Note that x−t 1 fˆh x = 1x − t ≤ hdFn t K h ∗ h x−t 1 E fˆh x = 1x − t ≤ hdFt K h ∗ h Furthermore,  x   x    sup Fx − Fh x ≤ sup ft − E fˆh t dt + sup E fˆh t − fˆh t dt x∈∗

x∈∗

0

x∈∗

0

(B.2) The first term on the right-hand side of the latter inequality can be estimated from (4.25). For the second term we get  x    x  t −y  1 K 1t − y ≤ h E fˆh t − fˆh t dt = sup sup h x∈∗ 0 h ∗ x∈∗ 0  d Fy − Fn ydt  x  ≤ sup K u dFt − hu − Fn t − hu dt u≤1 x∈∗ 0  x Ft − h − Fn t − h − Ft + h ≤ C sup x∈∗ 0

− Fn t + hdt

≤ 2C sup Fx − Fn x x∈∗

PROOFS OF CHAPTER 4

255

From (4.25), (4.29), (B.1) and (B.2) we get sup Fn x − Fh x ≤ 2C + 1 sup Fx − Fn x + 2 h2 K1 /2

x∈∗

(B.3)

x∈∗

since K1f  ≤ 2 K1 . Assume that sup Fn x − Fx ≤ 2 2C + 1−1 n−

(B.4)

x

Hence, for any solution h∗ of (4.28) we get, from (B.1) and (B.3), n− /2 ≤ 2 h2∗ K1 /2 Hence, it follows that h∗ ≥ 1 n−/2 

(B.5)

where 1 = 2 K1 −1/2 . We shall now prove that h∗ ≤ 2 n−/2 . For this purpose, we consider the auxiliary function  1 Ix h =

Fx − hy − Fx − Fn x − hy − Fn xKydy −1

Applying Taylor’s expansion to Fx − hy up to the term of order h2 , we get for any x that 

1

Fx − hy − FxKydy =

−1

h2  1 2 ′ y f hy Kydy ≥ h2 G 2 −1

1 where G = 1 /2 −1 y2 Kydy = K1 1 /2 is a positive constant. Assume (B.4). It follows that  1 Ix h ≤ Fx − hy − Fn x − hyKydy + Fx − Fn x −1

≤ 2C + 1−1 n−

(B.6)

Since h is selected from (4.28), we have, from (B.4) and (B.6),  1 h2∗ G ≤ sup Fx − hy − FxKydy x

−1

≤ sup Ix h + sup x

x

≤ 2/2C + 1 n−



1

−1

Fn x − hy − Fn xKydy

as n → . Hence, it follows that h∗ ≤ 2 n−/2 

(B.7)

256

PROOFS OF CHAPTER 4

where 2 = 2 2C + 1K1 1 −1/2 . One can see that assumption (B.4) leads to (B.5) and (B.7). From (7.5) it follows that 1 − P1 n−/2 ≤ h∗ ≤ 2 n−/2  < Psup Fx − Fn x > n− /22C + 1 x

 ≤ 2 exp −n1−2 /22C + 12 

Proof of Theorem 5. Suppose that any solution h∗ of (4.28) obeys the conditions h∗ > 2 n−/2 and h∗ < 1 n−/2 . Then from the assertion of Theorem 4 and (4.27) we get   2   MSEfˆh∗  ≤ 41 K1f x /4 n−2 + n−1+/2 fxK ∗ /2 + O n−1 This implies that     

2 P MSEfˆh∗  > 41 K1f /4 n−2 + n−1+/2 fxK ∗ /2 + cn−1 < 1 − P1 n−/2 ≤ h∗ ≤ 2 n−/2 

for some positive constant c. From (4.30) it follows that   PMSEfˆh  > c∗ n−4/5  ≤ 2 exp −n1/5 / 22C + 12 = n  when  = 2/5. The series  n=1 n converges and, by the Borel–Cantelli lemma, the assertion of the theorem holds.

Proof of Theorem 6. The proof is similar to the proof of Theorem 3. Suppose that h∗ → 0 as n → . This implies that, for any integer N > 0 ∃n > N such that h∗ = h∗ n > H∗ , where H∗ is some positive constant. We shall prove that, for such h∗ , sup− 2. Then, we have 

K3 4 −4/9 A ˜ x n < Ph∗ > n−1/m+1  P E f xh1  h∗  − fx > 24  ≤ 2 exp −2n1−2m+1/9 = n  Since the series  n=1 n converges, the assertion of the theorem holds by the Borel–Cantelli lemma.

Proof of Corollary 1. Denote K2∗ = K 2 tdt. From (B.11) and since EZ · fˆ A xh∗  = 0, the variance of f˜ A xh1  h∗  is     var f˜ A xh1  h∗  = var fˆ A xh∗  + c2 nh∗ −1 + onh∗ −1   = nh∗ −1 c2 + fx3/2 K2∗ + onh∗ −1  (B.13)

PROOFS OF CHAPTER 4

259

 From Theorem 7 it follows that h∗ = O n−1/9 if  = 9/m + 1 and m < 3 5. Hence, from (B.12) and (B.13) we have that  MSEf˜ A xh1  h∗  = K3 /242 h8∗ x2 + nh∗ −1 c2 + fx3/2 K2∗ + oh8∗  ∼ n−8/9 

as n → , if a maximal solution h∗ of (4.31) has order n−1/9 . Proof of Theorem 9. From (2.12) and (2.14) we obtain  1   2 2 1  j aj −  j fˆprn x − fx dx = 2 j=1 0

Let N be an arbitrary integer. Then

  N    2 1  2 1  2 1 (B.14) j aj − j = j aj − j +  a − j 2 j=1 2 j=1 2 j=N +1 j j 2 2 N  N    a j − j j2k+2 ≤ j2 + 2k+2 2k+2 j j 1 + 1 + j=1 j=1  2     aj + + j2 2k+2 j=N +1 1 + j j=N +1

We estimate each term on the right-hand side of this inequality. For the first term, we have 2 2 N  N   2   aj − j 1 n2  (B.15) a −  ≤ 2 ≤ sup j j 2k+2 2k+2  1≤j≤N j=1 1 + j j=1 1 + j

where

 n = sup aj − j 1≤j≤N

Since fx ∈ ℘ holds, according to Fikhtengol’ts (1965), for its Fourier coefficients we have the inequality j  ≤ 2Vk /j k+1 

j = 1 2 

(B.16)

where Vk is the variation of the function f k x. Therefore, for the second term on the right-hand side of (B.14), we have  2 2  N N   j2k+2 j2k+2 1 2 2 j ≤ 4Vk 2k+2 2k+2 2k+2 j j j 1 + 1 + j=1 j=1  2 N  jk+1 2k+2 2 = 4Vk  2k+2 j=1 1 + j

260

PROOFS OF CHAPTER 4

< 4Vk2 2k+1 1 + /2 < 8Vk2 2k+1

To estimate the third term, we take into account that aj  ≤ 2: 2 2       aj aj 4 ≤ ≤ 2k+2 2k+2 4k+4  4k + 3N 4k+3 j=N +1 1 + j j=N +1 j From (B.16), we have for the fourth term  

j=N +1

j2 ≤

4Vk2 2k + 1 N 2k+1

Drawing these results together, we obtain fˆpr x − fx2 ≤ 2

n2 4 + 8Vk2 2k+1 + 4k+4   4k + 3N 4k+3

+

4Vk2 2k + 1 N 2k+1

Since N is an arbitrary number, we take N = n1/k+1 . By the assumption of the theorem, it follows  = n−1/2k+2 . Therefore 2n2 1/2k+2 n + c1 n−2k+1/2k+2 + c2 n−2k+1/k+1     where c1 = 8Vk2  2k+1 , c2 = 4 1/  4k+4 4k + 3 + Vk2 /2k + 1 . Thus fˆpr x − fx2 ≤

n2k+1/2k+2 ˆ pr 2nn2 c c f x − fx2 ≤ + 1 + 2 n−2k+1/2k+2 = An + Bn  ln n  ln n ln n ln n

where An =

2nn2   ln n

 Bn = ln n−1 c1 + c2 n−2k+1/2k+2

For sufficiently large n, Bn ≤ 1 and An is a random variable. If An ≤ 8, then n2k+1/2k+2 ˆ pr f x − fx2 ≤ 9 ln n Consequently,  n2k+1/2k+2 ˆ pr 2 f x − fx > 9 < PAn > 8 P ln n

We estimate the right-hand side using Hoeffding’s inequality (Petrov, 1975). According to this inequality,  Pn > k < 2N exp −nk2 /8 (B.17)

PROOFS OF CHAPTER 4

Then



PAn > 8 = P n >



Since k ≥ 1, we have

8 ln n 2n  

n=1



< 2n

1/k+1

261

   ln n exp − = 2n1/k+1−/2 2

n1/k+1−/2 < 

and, according to the Borel–Cantelli lemma the first assertion of Theorem 9 holds. In order to prove the second assertion, instead of (B.15) one has to use the estimate  2  N N 4/n E2 X  −  2 /4   E aj − j 1 j j 2 = 2   j=1 1 + j2k+2 j=1 1 + j2k+2 ≤

N c3   n j=1

1

1 + j2k+2

2 ≤ 2

c3  n

where c3 is some constant. Then Theorem 9 is proved. For the proofs of Theorems 10 and 11 we need the following lemmas, in each of which we assume that X1   Xn is a sample of i.i.d. r.v.s with PDF fx ∈ ℘.

Lemma 6 If in estimate (2.14) we have  = G/n1/2k+3 , where G is some constant, then we have the inequality  Pfˆpr x X n  > B < 2n1/2k+1 exp −n2k−1/2k+1  √ where B = 2 + 4Vk + 2 2.

Lemma 7 For any N > 0, we have the inequality  1 2  1 + 2 4 Fn x − F  x2 dx < 4Vk2  2k+1  2k+3 1 + 2 + n + 2   N 0 Lemma 8 Assume that the regularization parameter  > 0 in the estimate of the PDF fˆpr x X n  can be found from (4.34). Then we have the inequality ⎞2 ⎛ N  aj j2k+2 2 2  ⎝  ⎠ ≤  n j=1 j 1 + j2k+2 where  > 0 (see (4.35)).

Proof of Lemma 6. obtain

From expression (2.14) for the estimation of fˆpr x X n  we fˆpr x X n  ≤ 1 +

  j=1

j aj 

262

PROOFS OF CHAPTER 4

We divide the sum on the right-hand side of this inequality into a finite sum of N terms and the remainder starting from the N + 1th index (N is an arbitrary integer). From (B.16) and from the fact that j  ≤ 1, aj  ≤ 2, j = 1 2 , we obtain   j=1

j aj 
2 2 We estimate the right-hand side by Hoeffding’s inequality. From (B.16) we obtain √ √  PKN > 2 2 = Pn > 2 2n−1/2k+1  < 2n1/2k+1 exp −n2k−1/2k+1  and the assertion of the lemma holds.

Proof of Lemma 7. manner:

F  x for estimate (2.14) is determined in the following

F  x = x +

 aj 1  sinjx   j=1 j 1 + j2k+2

(B.18)

The expansion of the empirical DF

Fn x =

n 1  x − Xi  n i=1

into a sine Fourier series on [0,1] has the form Fn x − x ∼

 aj 1 sinjx  i=1 j

(B.19)

PROOFS OF CHAPTER 4

263

where aj are the same coefficients as in (2.14). The Fourier series of the function Fn x does not converge to it on the entire segment [0,1]. As is known, at the points of discontinuity of the first kind Xi , i = 1  n, the series converges to the value Fn Xi + 0 + Fn Xi − 0 2

(Fikhtengol’ts, 1965). Then  a j2k+2 1  sinjx  j  i=1 j 1 + j2k+2 ⎛ ⎞2 2k+2  1   j a 1 ⎝  j ⎠ Fn x − F  x2 dx = 2 2 i=1 j 1 + j2k+2 0

Fn x − F  x ∼

(B.20)

Let N be an arbitrary integer. We divide the sum on the right-hand side of (B.20) into a finite sum of N terms and the remainder starting from the N + 1th index. We estimate each of the obtained terms: ⎛ ⎞2 ⎞2 ⎛ N N  aj j2k+2 j j2k+2 1 ⎝  ⎠ ≤ ⎝  ⎠ 2 i=1 j 1 + j2k+2 i=1 j 1 + j2k+2 N  2

+ n

i=1

⎛

j2k+2

⎞2

⎝  ⎠ j 1 + j2k+2

(B.21)

From (B.16) we obtain the inequality ⎛ ⎞2 ⎞2 ⎛ 2k+2 N N   j j2k+2 j 2k+2 ⎝  ⎝   ⎠ ≤ 4Vk2  ⎠ i=1 i=1 j 1 + j2k+2 j 1 + j2k+2

(B.22)

We estimate the second sum on the right-hand side of (B.21): ⎛ ⎞2 2k+2 N  j  ⎠ < n2  1 + 2 n2 ⎝  2k+2 i=1 j 1 + j

(B.23)

< 4Vk2 2k+3 1 + 2

  Since aj  ≤ 2, j = 1 2 and j2k+2 / 1 + j2k+2 < 1, we have ⎛

⎞2   aj j2k+2 1 4 ⎝  ⎠ ≤ 4 < 2 2k+2 j N j=N +1 j=N +1 j 1 + j  

(B.24)

264

PROOFS OF CHAPTER 4

From (B.20)–(B.24) we have  1 2  1 + 2 4 Fn x − F  x2 dx < 4Vk2  2k+1  2k+3 1 + 2 + n + 2   N 0 Then the lemma is proved. Proof of Lemma 8.

From (4.33) it follows that  1  Fn x − F  x2 dx ≥ n 0

From this and from (B.20) the assertion of the lemma follows. Proof of Theorem 10.

Let fˆpr x X n  ≤ B

(B.25)

Then from (4.34) and Lemma 8 for  = G/n1/2k+3 , we obtain the inequality  1 Fn x − F  x2 dx  ˆ 2n < maxB n 0    1/2k+3  G 2 2k+1 G < maxB n 4Vk  1 + 2 n n    1/2k+3   1/2k+3 4 G G 2 = n  1 + 2 + 2 + n n n  N Since N is arbitrary, we select N = n2 . We assume that  n ≤ 5 ln n/n (B.26)  2 2k+1 Then for G = / 8Vk  maxB  and sufficiently large n, the quantity n <  and  ˆ 2n < 

(B.27)

On the other hand, inequality (4.33) holds by assumption. This means that for  → , according to (4.33), (B.18) and (B.19) we have   1   aj 2 n  x − Fn x2 dx =  ˆ 2n → n ≥  (B.28) 2 2 j=1 j 0

Thus, under the conditions (B.25) and (B.26), from (B.27), (B.28) and from the continuity of  ˆ 2n with respect to  it follows that there exists  ≥ G/n1/2k+3 such 2 that  ˆ n = . In other words, if  is the largest value of the smoothing parameter such that  ˆ 2n = , then      1/2k+3  G ln n P < + Pfˆpr x X n  > B < P n > 5 n n

PROOFS OF CHAPTER 4

From (B.16) we obtain  

ln n P n > 5 n



265

 < 2n−9/8 + 2n1/2k+1 exp −n2k−1/2k+1

From this it is clear that, starting from some n, the assertion of the theorem holds. Proof of Theorem 11. We take a sample of sufficiently large size n. We consider (B.14). From Lemma 8 we obtain ⎞2 ⎛  2 2k+2 N N   j a j2k+2 j  ⎠ j2 j2 ≤ 2 ⎝  2k+2 2k+2 j 1 + i=1 i=1 j 1 + j  2 2k+2 N  j 4 2  2 + 2n2 ≤ N + 2n2 N 2k+2 n 1 + j i=1

From Lemma 8 we also obtain  

i=N +1



aj 1 + j2k+2

2

⎛

⎞2 aj j2k+2 j2 ⎝  ⎠ = j2k+4 i=N +1 j 1 + j2k+2  


8 c1 + c2  < PHn > 8 c1  + P < G/n1/2k+3 

In the same way as in Theorem 9, we estimate the first term on the right-hand side by Hoeffding’s inequality (Petrov, 1975): √  PHn > 8 c1  = Pn > 2 2n−k+1/2k+3  < 2n1/2k+3 exp −n1/2k+3

We estimate the second term on the right-hand side by Theorem 10. Now we consider the case when  is obtained in accordance with part two of the 2 method (4.33)–(4.35):  = n−1/2k+2 . Then by the proof of Theorem 9 we obtain that, for sufficiently large n, Pn2k+1/2k+3 fˆpr x X n  − fx2 > 9 < 2n1/k+1−/2 Therefore, if  is obtained by the 2 method, then we have the inequality

Pn2k+1/2k+3 fˆpr x X n  − fx2 > max8 c1 + c2  9  1/2k+3  1/2k+3  ≤ min 2n exp −n + 3n−9/8  2n1/k+1−/2 = n  The series  n=1 n converges and, by the Borel–Cantelli lemma, the assertion of Theorem 11 holds.

Appendix C

Proofs of Chapter 5

Proof of Theorem 12. Consider equation (5.4). In order to estimate the first term on the right-hand side, we note that Epˆ i gˆ i x = pi Kh ∗ gi x where the asterisk denotes convolution. Hence  1 qi T −1 xpi gi x − Epˆ i gˆ i xdx 0    h  1   −1 qi T x pi gi x Kh tdt − pi Kh ∗ gi x  dx =  0

≤

≤ =



0

−h

1

qi T −1 x

sup x∈01u≤1

sup x∈01u≤1



1

−1

pi gi x − uh − pi gi xKududx 1

gi x − uh − gi x



gi x − uh − gi x



0 1 0

qi T −1 x



1

Kududx −1

qi T −1 xdx

Since the PDF gi x is triangular, that is, gi x ≃ 1 − x1x ∈ 0 1, we have gi x − uh − gi x ≃ uh Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

268

PROOFS OF CHAPTER 5

By assumption (5.5),  1 qi T −1 xpi gi x − Epˆ i gˆ i xdx = Oh 0

We now estimate the second term on the right-hand side of (5.4). For the objects in the ith class, we have n x−y  1  j pˆ i gˆ i x = 1j = ipˆ i  1 ≤ i ≤ M K nh j=1 h where j are the labels of the objects. Therefore, pˆ i gˆ i x can be expressed as x−t 1 1 pˆ i dGin t K pˆ i gˆ i x = h 0x−t≤h h

where Gin t is the empirical DF constructed from transformed observations yj = Txj  j = 1  n, corresponding to the ith class. Obviously, x−t 1 1 Epˆ i gˆ i x = pˆ i dGi t K h 0x−t≤h h

where Gi t is the DF of yj = Txj  j = 1  n, if O ∈ Pi . Hence we have  1 qi T −1 xpˆ i gˆ i x − Epˆ i gˆ i xdx 0    1 x−t  1  1  −1 i i K q T x ≤ dGn t − G t dx h 0  i h 0x−t≤h   1   1  i i qi T −1 x  dx ≤ KudG x − uh − G x − uh n   0

≤ C1

−1



1

Gin x − h − Gi x − h − Gin x + h − Gi x + hdx

0

≤ 2C1 sup Gin x − Gi x x

Substituting into (5.4), we obtain   qi xpi fi x − pˆ i fˆi xdx < 2C1 sup Gin x − Gi x + Oh 0

x

Since h = n , from (5.3) we obtain −

˜ EB  − B nd < 2C1 M + 1nd sup Gin x − Gi x + M + 1Ond−   x

Let Bn denote the first term on the right-hand side of this inequality. For sufficiently large n and d < , the second term is less than one. Hence, if Bn ≤ 1, then ˜ EB  − B  ≤ 2. Therefore, nd  Pnd ˜ EB  − B  > 2 < PBn > 1 

PROOFS OF CHAPTER 5

269

To estimate PBn > 1 we use Prakasa Rao’s (1983) result, Psup Gin x − Gi x > ≤ 2 exp−2 2 n x

which is valid for any i. Hence  PBn > 1 ≤ 2 exp −

 n1−2d  2M + 12 C12

Let Hn d denote the right-hand side of this inequality. If d < min0 5 , then the series  n=1 Hn d converges and, by the Borel–Cantelli lemma, we arrive at the assertion of the theorem. Proof of Theorem 13. Let us consider (5.4). By the Hölder inequality,  1     qi T −1 xpi gi x − pˆ i gˆ i xdx   0

≤



1

0

qi T −1 x2 dx

Let us assume that



0

 21 

1

0

1

pi gi x − pˆ i gˆ i x2 dx

2

pi gi x − pˆ i gˆ i x dx

 21

 21

(C.1)



≤ n−d 

Then, from (5.3)–(5.5) and (C.1), for C1 > 0, we obtain ˜ EB  − B  ≤ M + 1C1  nd  Hence, ˜ EB  − B  > M + 1C1 < P Pn  d

≤P

 

1 0

2

pi gi x − pˆ i gˆ i x dx 2

sup gi x − gˆ i x > n

x∈01

 21

−2d

>n

−d





= Pˆgi x∗  > gi x∗  + n−d + Pˆgi x∗  < gi x∗  − n−d 

where x∗ ∈ 0 1. Let z = gˆ i x and gi = gi x. By the assumptions of the theorem, the polygram has an asymptotic distribution (as n → )



 √  gi /z5/2 L gi /z2 1 1 − gi /z2 pz = √ + 1 + O1/ n  exp − 2 gi /Lz m1 − gi /mz gi 2/L

270

PROOFS OF CHAPTER 5

(Tarasenko, 1976). Therefore, gi /z5/2 L   · 2 gi +1/nd gi



  √  1 1 − gi /z2 L gi /z2 · exp − + 1 + O1/ n dz 2 gi /Lz m1 − gi /mz

Pˆgi x∗  > gi x∗  + n−d =



Applying the substitution u = gi /z, we obtain

   d  √  Lu2 L  gi /gi +1/n  √ 1 L1 − u2 + 1 + O1/ n du u exp − 2 0 2 u m1 − u/m   √  √ L ≃ L exp−L/n2d  ≃ n exp−n −2d  ≤ L/2 exp − 2d −d n gi + n gi

=



This can also be proved for Pˆgi x∗  < gi x∗  − n−d . Let Hn d  denote the right-hand side. Since 0 < d < /2, we find that n=1 Hn d  converges and, by the Borel–Cantelli lemma, we arrive at the assertion of the theorem.

Appendix D

Proofs of Chapter 6

Proof of Theorem 14.1 (6.7) we have

  Let us find first the distribution of log xpw /xp . From

−1/ˆ −2/ˆ −1/ˆ −1/ˆ ˆ = ˆ log1 + Xn−k ˆ n−k ˆ log c + Xn−k  ≃ X + oXn−k 

(D.1)

−1

We use the relation Xi =d F exp−Si  1 ≤ i ≤ n, where F x = 1 − Fx Si are order statistics, corresponding to the sample of size n of independent exponentially distributed r.v.s with unit expectation, and exp−Si  are order statistics of the sample of independent uniform distributed r.v.s on (0,1). For distributions of Pareto type (6.3) we have   c −1 d xp = F p = 1 + dc− p + op  (D.2) p   c 1 + dc− exp− Sn−k  + oexp− Sn−k  Xn−k =d exp−Sn−k  (D.3) log Xn−k =d log c + Sn−k  + dc− exp− Sn−k  + oexp− Sn−k  (D.4) 1

Taken from Performance Evaluation, 62(1–4), pp. 178–192, High quantile estimation for heavy-tailed distributions, Markovich NM, Section 3, © 2005 Elsevier. With permission from Elsevier.

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

272

PROOFS OF CHAPTER 6

(all equations are satisfied in probability). From (6.5) and (D.1)–(D.4) it follows that  w xp log = log Xn−k − log xp + ˆ log an (D.5) xp     n  k d + ˆ −  log =  Sn−k − log k np + dc− exp− Sn−k  − p  + oexp− Sn−k  + op 

From the Rényi representation Sn−k =

d

n−k  j=1

zj = n−k + Tn−k  n−j +1

(D.6)

where zj are independent exponentially distributed r.v.s with unit expectation, and zj − 1  n −j +1 j=1     n−k n   1 1 n 1

n−k = = = log +O  n − j + 1 j k + 1 n j=1 j=k+1 Tn−k =

d

n−k 

for 1 ≤ k < n, we get the expectation and the variance of Sn−k :     n 1 ESn−k = log +O  k+1 n   1 1 1 1 1 varSn−k = +   + 2 ≃ − +O  2 k + 1 n k+1 n k

(D.7)

Furthermore, we have

exp− Sn−k  = exp− ESn−k  exp− Sn−k − ESn−k  −      n n exp − Sn−k − log ≈ k+1 k+1      k+1 n n = − Sn−k  + o log − Sn−k  1 +  log n k+1 k+1 (D.8) Taking into account (D.8), we consider equation (D.5). We obtain that  w  

  n  xp k + 1  − d log  1 −  dc = Sn−k − log xp k n 

   k + 1  k −  + dc −p  + ˆ −  log np n

(D.9)

PROOFS OF CHAPTER 6

273

It is known that √

  k ˆ −  →d N 0  2   n  √  →d N 0 1 k Sn−k − log k

see Embrechts et al. (1997, p. 341) or Beirlant et al. (2004, p. 109). Thus   

  2 2 k k  d log ˆ −  → N 0 log  np np k     n 

2 + a →d N a  ∀ a d

Sn−k − log k k

(D.10)

Furthermore, we have   2 2

 w xp k 

2 d + log  → N a log xp k np k where a = dc

−



k+1 n



−p







=  1 −  dc



−



k+1 n





Thus,



  log xpw /xp − a

d 1/2 → N 0 1 

2 /k + logk/np2  2 /k

We now find the DF for logxpc /xp . Note that

(D.11)

 ˆ −1/ˆ −2/ˆ −1 ˆ = log 1 + Xn−k ˆ log c ≈ Xn−k  + Xn−k

We take, for simplicity,

−1

Xn−k =d F ∗ exp−Sn−k  −1 where F ∗ x = x−1/ , that is, Xn−k =d exp−Sn−k . From (D.8) we have            k+1  n n −1 Xn−k =d 1 +  log − Sn−k + o log − Sn−k  n k+1 k+1 (D.12)

From (D.10) it follows that −1 Xn−k

d

−→ N



k+1 n



2  k



k+1 n

2



274

PROOFS OF CHAPTER 6

Thus from (6.9), (D.9), and (D.12) we obtain  c    

  n  xp k + 1 k + 1 log  1 −  dc− = d Sn−k − log − xp k n n

      k+1  k + 1  k+1  −  + + dc −p − n n n   k ˆ −  + log np    2 2

 k + 1  ∗2 k  d → N a+ + log   n k np k   and it follows that where ∗ = −  k+1 n logxpc /xp  − a + k + 1/n  d 1/2 → N 0 1   2

∗2 /k + log k/np  2 /k

The theorem follows from (D.11) and (D.13).

(D.13)

Appendix E

Proofs of Chapter 7

Proof of Lemma 4.

We denote by ·C the norm in the space C0 xa . Then,

y − yn C = sup− ln1 − Fx + ln1 − Fn x + ∗ x x

   1 − Fn x + ∗ x    = supln  1 − Fx x     Fx − Fn x + ∗ x  = supln 1 +  1 − Fx x

Let us introduce the sets

A = x 0 ≤ Fx − Fn x + ∗ x

Then for x ∈ A,

and

B = x 0 ≤ Fn x − ∗ x − Fx

    Fx − Fn x + ∗ x   y − yn C = supln 1 +  1 − Fx x∈A   ∗ Fx − Fn x +  x = sup ln 1 + 1 − Fx x∈A   Fn x − ∗ x − Fx = ln 1 + sup 1 − Fx x∈A   Fn x − ∗ x − Fx  = ln 1 + sup 1 − Fx Fx≤a

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

276

PROOFS OF CHAPTER 7

and for x ∈ B,

    Fx − Fn x + ∗ x   y − yn C = supln 1 +  1 − Fx x∈B      F x − ∗ x − Fx  = supln 1 − n  1 − Fx x∈B   F x − ∗ x − Fx = − ln 1 − sup n 1 − Fx x∈B   Fn x − ∗ x − Fx = − ln 1 − sup  1 − Fx Fx≤a

Hence,  Fn x − ∗ x − Fx  y − yn C ≤ max ln 1 + sup 1 − Fx Fx≤a   Fn x − ∗ x − Fx − ln 1 − sup 1 − Fx Fx≤a   Fn x − ∗ x − Fx = n = − ln 1 − sup 1 − Fx Fx≤a  

Proof of Theorem 18. Let us consider the operator equation (7.2). Let g be the regularized estimate and g be the solution of (7.2). The proof is based on the Theorem 17 in Stefanyuk (1986). We use Prakasa Rao’s (1983) inequality, according to which

(E.1) P supFn x − Fx > ≤ 2 exp−2n 2  x

for sufficiently large n, and obtain with regard to the inequality of Lemma 4 that ⎧ ⎫ ⎞ ⎛

supFn x − ∗ x − Fx ⎨ yn − yC √ ⎬ x ⎠ ⎝ P > c > c ≤ P − ln 1 − √ 1 1 ⎩ ⎭

1−a

√ 2 ≤ 2 exp−2n 1 − a1 − exp−c1  − 1  = Pn  a  Since is defined so that = n−→n→ 0 n=1 Pn  a < holds. In view of the fact that the operator A is defined precisely, we get from (7.7) for the solution of equation (7.21) that

yn − yC

> c1  P  h − hC >  ≤ P 

√

Then Theorem 18 follows from the Borel–Cantelli lemma.

PROOFS OF CHAPTER 7

Proof of Theorem 19.

277

The inequality

h x − h x A yn  ≤ h x − h x A y

(E.2)

+ h x A y − h x A yn 

where the function h x A y is the solution of the equation

h + A∗ Ah = A∗ y

> 0

is valid for the error of the regularized estimate of (7.21). To represent the solution explicitly, we make use of the method of E. Schmidt (Ivanov et al., 1978). We have h t A y = Hence,

 i=1

 i ai c  t =  t 2 i i 1 + i 1 +

2i i i=1

   

2i   a  t h t − h t A y =    i=1 1 + 2i i i 

by virtue of (7.27). We apply the Parseval equality and obtain 2

h x − h x A y =

xa 0

2

h t − h t A y dt =

 i=1



2i a 1 + 2i i

2



We denote R =  I + A∗ A−1 A∗ and obtain the estimate of the second term on the left-hand side of (E.2):     h x A y − h x A yn  =  I + A∗ A−1 A∗ y − yn  ≤ R  n (E.3) It follows from (7.25) and (7.28) that   i=1

2i 1 + 2i

2

a2i

≤



For an arbitrary integer N , ⎛   2 ⎞2 i

⎜ xa 1 ⎟ ⎝  2 ⎠ 2k+1 i i=1 1 + xi

Vk xa

2

 i=1

⎛ ⎜ ⎝

  2 ⎞2

1+

i xa

⎟  2 ⎠ i xa

1 i2k+1



(E.4)

a

⎛   2 ⎞2   2 ⎞2 i N

xi   xa 1 1 a ⎟ ⎟ ⎜ ⎜ + = ⎝ ⎝  2 ⎠ 2k+1  2 ⎠ 2k+1  i i i i i=N +1 i=1 1+ x 1+ x ⎛

a

a

278

PROOFS OF CHAPTER 7

We estimate the first term on the right-hand side of (E.4). For k ∈ 0 1 and N ≤ √1 , we get N  i=1

⎛ ⎜ ⎝

  2 ⎞2

1+

i xa

⎟  2 ⎠ i xa

For k ≥ 2, we get

1 i2k+1

=



√

xa

2k+2

 2k+1  =O 2 

⎤2  √ 2 i

⎥ xa ⎢ ⎥  √ 2   √ k+1 ⎦ ⎣ i i i=1 1+ x

x ⎡

N ⎢ 

a

a

⎛  2 ⎞ 2 N

xi  # 2$ 1 a ⎟ ⎜ ⎝  2 ⎠ 2k+1 = O  i i=1 1 + xi a

Turning to the second term on the right-hand side of (E.4), we have that ⎛  2 ⎞2 i

  ⎜ xa 1 1 1 ⎟ ≤  ⎝  2 ⎠ 2k+1 ≤ 2k+1 2k + 1 N 2k+1 i i=N +1 i i=N +1 1 + xi a

Since N is an arbitrary number, we take % 2 ( & '  2k+1 1 1  N = min √ 

where · is the integer part of a real number, and obtain that  # $ O 2k+1/2  k = 0 1 2

# $ (E.5) h x − h x A y = O 2  k ≥ 2    We now estimate R  (see (E.3)). Let hx = i=1 hi i x ∈ L2 0 xa . Then, R h =

 i=1

i h  t 1 + 2i i i

We obtain from the definition of norm and the Parseval equality that    2  2   i 2 2 R  = sup hi hi ≤ 1  1 + 2i i=1 i=1

where the supremum is taken over sequences i  and hi . The function g = √  at  = √1 reaches the maximum 1/2 . Hence, 1+ 2   1 R  ≤ √  2

PROOFS OF CHAPTER 7

279

Then, 1 h x A y − h x A yn  ≤ √ n 2 From this and from (E.2) and (E.5), we obtain  n c 2k+1/4  h x − h x A yn  ≤ √ + 2 c 

k ∈ 0 1 k ≥ 2

where c is a constant independent of n. Let = n− ,  > 0, be a constant. Then, for some  > 0 and k ∈ 0 1, n h x − h x A yn  ≤ An + Bn  

where An = cn−2k+1/4 and Bn = nn 2 + /2, while for k ≥ 2, n h x − h x A yn  ≤ Cn + Bn  where Cn = cn− . Let us consider the case k ≥ 2. Since  ≥ , for sufficiently large n we have that Cn ≤ 1 and Bn is a r.v. If Bn ≤ 1, then n  x −  x A yn  ≤ 2 Consequently, ) * P n  x −  x A yn  > 2 < P Bn > 1

The right-hand side is estimated using inequality (E.1). By Lemma 4,     Fn x − ∗ x − Fx − 2 − P Bn > 1 = P − ln 1 − sup > 2n 1 − Fx Fx≤a

# # $$  = P supFn x − ∗ x − Fx > 1 − exp −2n− 2 − 1 − a x

 $$ $2  # # #  =  n  ≤ 2 exp −2n 1 − a 1 − exp −2n− 2 − − 1

 Since  < 1 − 2, the series n=1  n < , and the assertion of the theorem is valid by the Borel–Cantelli lemma. We now turn to the case k ∈ 0 1. Since  ≥ 4/2k + 1 or −

2k + 1 ≤ 0 4

An ≤ C for sufficiently large n and some constant C. If Bn ≤ 1, then n h x − h x A yn  ≤ 1 + C

(E.6)

280

PROOFS OF CHAPTER 7

Therefore, ) * P n h x − h x A yn  > 1 + C < P Bn > 1

The assertion of the theorem follows the same arguments as given before for estimating P Bn > 1. Proof of Lemma 5. Let us estimate the inaccuracy of defining the operator An g − AgC = supAn g − Ag ≤ n g x ∈ 0 xa , for equation (7.31). We have x

    x   Ag − An gC ≤ sup Iy − In2 ygx − ydy x   0     x    + sup In2 yHn1 y − IyHygx − ydy x   0

(E.7)

Let us estimate the first term on the right-hand side of (E.7):     x x       sup Iy − In2 ygx − ydy ≤ sup Iy − In2 ygx − ydy x x   0 0 x     ≤ sup Ix − In2 x gx − ydy x

0

  = supIx − In2 x

(E.8)

x

since gx is the PDF. Turning to the second term of the right-hand side of (E.7), since     In yHn y − IyHy = Hn yIn y − Iy + IyHn y − Hy 1 2 1 1 2     ≤ In2 y − Iy + IyHn1 y − Hy we obtain that

    x   sup In2 yHn1 y − IyHygx − ydy x   0

x   ≤ sup In2 yHn1 y − IyHygx − ydy x

0

    ≤ supIn2 x − Ix + supIxsupHn1 x − Hx x

x

x

(E.9)

PROOFS OF CHAPTER 7

Then from (E.7)–(E.9) it follows that     An g − AgC ≤ 2supIn2 x − Ix + supIxsupHn1 x − Hx x

x

281

(E.10)

x

In the notation of (7.34) we have      fn2 x fx  In x − Ix =  − (E.11) 2  1 − F x + ∗ x 1 − Fx  n2  $  #   fn x − fx 1 + Fn x − Fx + fx Fx − Fn x + ∗ x 2 2 2  ≤ 1 − aC ∗ Furthermore, it follows that

supIx = sup x

x

fx fx ≤ sup  1 − Fx x 1−a

Hence, from (E.10) and (E.11) we get the assertion of the lemma. Proof of Theorem 20. We denote supx = supx∈0xa  . Note that supx ∗ x = max1/n 1 − a = max . Hence, by virtue of the condition, it follows from Lemma 5 that   %     2     sup f x − fx 1 + sup Fn2 x − Fx An g − AgC ≤ 1 − aC ∗ x n2 x &      + max + supFn2 x − Fx + supHn1 x − Hx 1−a x x   + We fix the constants c1  c2 > 0 and c3 > 2 / C ∗ 1 − a min and denote

+   4   A= 

sup f x − fx ≤ c1 min  1 − aC ∗ x n2

+   B= 

supHn1 x − Hx ≤ c2 min  1−a x

  +   2   C= 

sup Fn2 x − Fx + max ≤ c3 min  1 − aC ∗ x

If the events A B and C occur simultaneously, then for any function gx we have + An g − AgC ≤ c1 + c2 + c3  min  Then,

, + ) * ) * ) * ¯ + P B¯ + P C¯  P An g − AgC > c1 + c2 + c3  min ≤ P A

It follows from (7.8) that

282

PROOFS OF CHAPTER 7



+ √ P An − A > c1 + c2 + c3   ≤ P supAn g − AgC > c1 + c2 + c3  min g∈D

  c1 1 − a C ∗ +  

min ≤ P sup fn2 x − fx > 4 x

  c 1 − a + + P supHn1 x − Hx > 2

min x

  c3 1 − a C ∗ +   + P sup Fn2 x − Fx >

min − max  (E.12) 2 x + + We denote c2 = c2 1 − a min / and c3 = c3 1 − a C ∗ min / 2  − max . Then we obtain from (E.1) that

  # $ P supHn1 x − Hx > c2 ≤ 2 exp −2n1 c2 2  (E.13) x

  # $   P sup Fn2 x − Fx > c3 ≤ 2 exp −2n2 c3 2 

(E.14)

x

By the to statistical regularization theory (see p. 184) the regularized estimates fn 2 x and yn 1 x converge to the true PDFs fx and yx with probability one under the conditions (7.32) on n assumed in the theorem for  = min1  2 . Hence, from (7.6) we get

P supyn 1 x − yx > 1 ≤ 2 exp −n1 1   (E.15) x

P



supfn 2 x − fx x



> 2 ≤ 2 exp −n2 2  

(E.16)

for some numbers N1 = N1 1  1  N2 = N2 2  2  and all n1 > N1  n2 > N2 . Here 1  2  1 and 2 are any positive numbers. + √ Let 1 = c1 1 − a C ∗ min /4 and 2 = c . It follows from (7.7) and (E.12)–(E.16) that   

y x − yx An − A n 1

C P g − gC >  ≤ P > c +P > c1 + c2 + c3 √ √

# # $ ≤ 2 exp −n1 1  + exp −n2 2  + exp −2n1 c2 2 $$ # + exp −2n2 c3 2 

Hence, for the chosen sequence = n we get the assertion of the theorem.

Proof of Theorem 21. For brevity, we denote g = g x g = gx h = h x h = hx G = G x G = Gx. It follows from G ≤ C and Gxa  = b < 1 that

PROOFS OF CHAPTER 7

283

     g   g − g + gG − G − Gg − g  g

  sup  h − hC =   1 − G − 1 − G  = x∈0x  1 − G  1 − G a C & % supx g 1 supG − G  (E.17) supg − g + ≤ 1−C x 1−b x

Additionally,

   x   

 G − GC = sup g  − g d  ≤ supg  − gxa  x  x  0

The theorem follows from this fact, (7.33), and (E.17) by virtue of the fact that gx is bounded.

Appendix F

Proofs of Chapter 8

Proof of Theorem 23.

By (8.5) we get for 0 ≤ t ≤ tmax k:

 k l ≤ sup sup Ht − Ht t

t

 

n=k+1

P tn < t + k max sup P tn < t − Fln t 1≤n≤k

t

Under the conditions of the theorem it follows from well-known results that   m−2  Qi t   t − n P n √ < t = t + + o n−m−2/2 (F.1) i/2 n i=1 n uniformly in t ∈ − , where Qi are expressions involving the PDF x = 2 −1/2 exp−x2 /2 of the standard normal DF x, the Hermite polynomials and semi-invariants of i (see Embrechts et al. 1997, Theorem 2.3.2, p. 85; Petrov, 1975). In particular, Q1 x = x

1 − x2 E 1 − 3 6 2

and, for m = 3,    tn − n 1 E 1 − 3 1 2 2 P exp−t /2 + o √ √ 1 − t  √ √ < t = t + n 6 3 n n 2

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

286

PROOFS OF CHAPTER 8

Hence, P tn < t is√ defined like the right-hand side of (F.1) with the replacement of t by t − n/ n. Then  n=1 P tn < t converges for m ≥ 3 and  

n=k+1

holds. If

c>0

P tn < t ≤ c

(F.2)

k max sup P tn < t − Fln t ≤  1≤n≤k

t

holds for any constant  > 0, then at t ∈ 0 tmax k,  k l ≤ c +  sup Ht − Ht t

follows. Hence,      P sup Ht − Ht k l > c +  < P k max sup P tn < t − Fln t >   1≤n≤k

t

t

The right-hand side may be estimated using the asymptotical estimate of the convergence rate of the empirical DF to the true DF (Prakasa Rao, 1983),     (F.3) P sup P tn < t − Fln t >  ≤ 2 exp −2ln 2  t

which is satisfied for sufficiently large l. Then it follows that    l 2  P sup Ht − Ht k l > c +  < 2 exp −2 3  = P l k k t

  Since k ∼ l  0 <  < 1/3, the series l=1 P l k converges at least for one  > 0, and by the Borel–Cantelli, the assertion of the theorem follows. Proof of Theorem 24.

Using (8.8) we have, for t ∈ 0 1,

 k l ≤ 1 − exp−k+1 exp sup Ht − Ht t

+ k max sup P tn < t − Fln t 1≤n≤k

t

and, for  > 0,  k l ≤l 1 − exp−k+1 exp l sup Ht − Ht t

+ l k max sup P tn < t − Fln t 1≤n≤k

t

Since k = c · l , where  < 1/3 − 2/3 0 <  < 05  > 0, for sufficiently large l and the corresponding c we get 

l 1 − exp−k+1 exp ≤ 1

PROOFS OF CHAPTER 8

287

therefore, if l k max sup P tn < t − Fln t ≤  1≤n≤k

t

for any constant  > 0, then it follows that  k l ≤ 1 +  l sup Ht − Ht t

Hence,      k l > 1 +  < P l k max sup P tn < t − Fl t >   P l sup Ht − Ht n 1≤n≤k

t

t

Using (F.3) we have     2  k l > 1 +  < 2 exp −2 l  P l sup Ht − Ht = P l k (F.4) k3 l  t

Since k = c · l and  + 15 < 05, the series  l=1 P l k converges at least for one  > 0, and by the Borel–Cantelli lemma, the assertion of the theorem holds. Proof of Corollary 2.

Hence, we have

Let the right-hand side of (F.4) be equal to 0 <  < 1:  l   2 2 exp −2 3  = k l

k ln /2  2l This gives the level of the confidence interval D = 1 + l− .  = kl

Proof of Theorem 25. sufficiently large n,



−

Since, for t ∈ 0 hck , expression (8.10) is valid for k

 

 



t Ptn < t ∼

√ n n=k+1 n=k+1



holds. The expansion on the right-hand side converges. Therefore,  n=k+1 Ptn < t < c follows, where c > 0 is a constant. The rest of the proof is similar to the proof of Theorem 23. For t ∈ a tmax k, a > 0 we have, from (8.11),  k+1  t y − − k+1 k+1 1 − t ct exp dy x0 y 1 − ℓtt  Ft  = sup = sup sup t 1 − Ft ℓtt− t t t t− ct exp x y dy y

Proof of Theorem 26.

0

The mean value theorem implies  t       y t t exp = dy = exp  ln  y x x x0 0 0

288

PROOFS OF CHAPTER 8

for some  ∈ x0  t. Hence,

k+1  − 1 − ctt−+ x0

k+1

sup t

Ft = sup 1 − Ft t

−

ctt−+ x0



Since x is nonpositive, − +  < 0. Then we have k+1  −+ − k+1 1 − c t k x inf max 0 Ft  = sup − 1 − Ft t cinf tmax k−+ x0

where

cinf = inf ct = t



(F.5)

if ct is a monotone decreasing function if ct is a monotone increasing function

ctmax k ca

Since ctmax k > c0  cinf ≥ minc0  ca = c∗ and the right-hand side of (F.5) is less than  k+1 − 1 − c∗ tmax k−+ x0  (F.6) − c∗ tmax k−+ x0 Assume that max i ≤ l 

(F.7)

i

where  > 1/  − ∗ . Then

tmax k ≤ l max i < l1+  i=1    l

This implies that (F.6) is less than or equal to  k+1 − 1 − c∗ l1+−+ x0 −

c∗ l1+−+ x0



Then from (F.5) we have k+1  − k+1 1 − c∗ l1+−+ x0 Ft sup l  ≤ l − 1 − Ft t c∗ l1+−+ x0

Since 1 + − +  < 0, for sufficiently large l the right-hand side of the latter inequality is less than or equal to 1 for k ≥ −A · l and  ≥ 0.1 Note that, for  > 0 and sufficiently large l A < 0 holds. Therefore, if l k max sup Ptn < t − Fln t ≤  1≤n≤k

1

(F.8)

t

One can obtain this result by taking the left-hand side of the latter inequality to be less than or equal to 1.

PROOFS OF CHAPTER 8

289

for any constant  > 0, it follows from (8.5) and (8.6) that  t k l  ≤ 1 +  l sup Ht − H t

Hence, from (F.7) and (F.8),

 t k l  > 1 +   Pl sup Ht − H n 1≤n≤k

t

t

+ Pmax i > l  i

follows. By the global property of the regularly varying r.v.s (see Embrechts et al., 1997, p. 38) we have as x → 

Pmax i > x ∼ lP i > x = lx− ℓx i

From the representation theorem (8.11) the following property of slowly varying functions follows (Mikosch, 1999): for every ∗ > 0, ∗

x− ℓx → 0

∗

and

x ℓx → 

as x → 

This implies that there exists T > 0 such that, for x > T , ∗

∗

x− ≤ ℓx ≤ x  Since ∗ <  it follows that ∗ −

Pmax i > l  ∼ l1+ i

as l → 

Using (F.3), we finally have ∗ −

 t k l  > 1 +  < l1+ Pl sup Ht − H t

  + 2 exp −2 2 l1−2 /k3 = P l k

(F.9)

Since k = dl and 0 <  < 1 − 2/3  > 1/ − ∗  hold, the series

 l=1 P l k converges at least for one  > 0, and by the Borel–Cantelli lemma, the assertion of the theorem holds. Proof of Corollary 3.

Hence, we have

Let the right-hand side of (F.9) be equal to 0 <  < 1:   ∗ l1−−  + 2 exp −2 2 l1−2 /k3 =   = kl





 k  − l1−−∗  − ln  2l 2

This gives the level of the confidence interval D = 1 + l− .

List of Main Symbols and Abbreviations

General guidelines In general, Greek letters represent parameters. X n = X1  X2      Xn  X1 ≤ X2 ≤    ≤ Xn R R+ Z n h AT rank A  and = 1/ = n

data sample order statistics real line nonnegative real line set of integers sample size bandwidth in kernel estimators transpose of matrix A rank of matrix A tail index and extreme value index regularization parameter

Let the functions fx and gx be defined on some set M and let a be a limit point of M.

fx ∼ gxx → a x ∈ M fx = ogxx → a x ∈ M fx = Ogxx ∈ M

denotes a function fx that satisfies limx→ax∈M fx/gx = 1 denotes a function fx that satisfies limx→ax∈M fx/gx = 0 denotes, for some constant C > 0, the inequality fx ≤ Cgx for all x ∈ M

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

292

LIST OF MAIN SYMBOLS AND ABBREVIATIONS

Probabilities P· E· var· bias· covX Y N   x fx gx Hx hx

probability measure expectation variance bias covariance between the r.v.s X and Y normal (or Gaussian) density with mean and variance  2 the distribution function of the standard normal distribution probability density functions renewal function hazard rate function

Functions Ax

x ℓx Kx Tx F −1 x  x x > 0 x+ = 0 x ≤ 0  1 t ≥ 0 t = 0 t < 0  1A = 1x ∈ A =

Pickands’ dependence function gamma function slowly varying function kernel function transformation function of the data inverse function indicator of positiveness indicator function of nonnegativity 1 0

x ∈ A otherwise

indicator function of the event (set) A norm of x

x

Spaces Ca b

ℓp ℓ2 Lp a b, p ≥ 1 H a b

space of all continuous real-valued functions defined on the closed interval a b with norm xC = maxa≤t≤b xt space of sequences x = x1  x2      xn      of real numbers  p 1/p such that xℓp =  < , p ≥ 1 k=1 xk   Hilbert space ℓp , p = 2, with a scalar product  x y = k=1 xk yk a xtp dt < space of functions with b  a the integral and norm xLp =  b xtp dt1/p Hölder space with norm   xH = sup x t + t∈ab

sup

t1 t2 ∈abt1 =t2

xt1 −xt2  t1 −t2 

LIST OF MAIN SYMBOLS AND ABBREVIATIONS

Signs ≃

asymptotic equality

Abbreviations ACF AMISE ARMA BISDN DF EPM EVI EVD GARCH GE GEV GPD HTML i.i.d. ITD MA MDA MIAE MISE ML MSE PDF POT PWM RF r.v. SMIL TCP UMTS WWW

autocorrelation function asymptotical mean integrated squared error autoregressive moving average process Broadband integrated services digital network distribution function elemental percentile method extreme value index extreme value distribution generalized autoregressive conditionally heteroscedastic process Gilbert–Elliott model generalized extreme value generalized Pareto distribution hypertext markup language independent and identically distributed inter-arrival time distribution moving average process maximum domain of attraction mean integrated absolute error mean integrated squared error maximum likelihood mean squared error probability density function peaks over threshold method of probability-weighted moments renewal function random variable Synchronized Multimedia Integration Language transmission control protocol Universal Mobile Telecommunications System World Wide Web

293

References Aas, K. and Haff, I.H. (2006) The generalized hyperbolic skew Student’s t-distribution. Journal of Financial Econometrics 4(2), 275–309. Abramson, I.S. (1982) On bandwidth estimation in kernel estimators – A square root law. Annals of Statistics 10, 1217–1223. Adler, R.J., Feldman, R.E. and Taqqu, M.S. (eds) (1998) A Practical Guide to Heavy Tails: Statistical Techniques and Applications. Birkhäuser, Boston. Ahrends, J.H. and Dieter, U. (1974) Computer methods for sampling from gamma, beta, Poisson and binomial distributions. Computing 12, 223–246. Aivazyan, S.A., Buchstaber, V.M., Yenyukov, I.S. and Meshalkin, L.D. (1989) Applied Statistics. Classification and Reduction of Dimensionality. Financy i statistika, Moscow (in Russian). Asmussen, S. (1996) Renewal theory and queueing algorithms for matrix-exponential distributions. In S.R. Chakravarthy and A.A. Alfa (eds), Matrix-Analytic Methods in Stochastic Models, pp. 313–341. Marcel Dekker, New York. Azoury, K.S. and Warmuth, M.K. (2001) Relative loss bounds for on-line density estimation with the exponential family of distributions. Machine Learning 43, 211–246. Balkema, A. and de Haan, L. (1974) Residual lifetime at great age. Annals of Probability 2, 792–804. Barndorff-Nielsen, O.E. (1977) Exponentially decreasing distributions for the logarithm of particle size. Proceedings of the Royal Society of London A 353, 401–419. Barron, A.R. and Sheu, C.-H. (1991) Approximation of density functions by sequences of exponential families. Annals of Statistics 19, 1317–1369. Barron, A.R., Györfi, L. and van der Meulen, E. (1992) Distribution estimation consistent in total variation and in two types of information divergence. IEEE Transactions on Information Theory 38, 1437–1454. Baxter, L.A., McConalogue, D.J., Scheuer, E.M. and Blischke, W.R. (1982) On the tabulation of the renewal function. Technometrics 24, 640–648. Beirlant, J., Dierckx, G., Goeghebeur, Y. and Matthys, G. (1999) Tail index estimation and exponential regression model. Extremes 2, 177–200. Beirlant, J., Goeghebeur, Y., Teugels, J. and Segers, J. (2004) Statistics of Extremes: Theory and Applications. Wiley, Chichester. Beran, J. (1994) Statistics for Long-Memory Processes. Chapman & Hall, New York.

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

296

REFERENCES

Berlinet, A., Vajda, I. and van der Meulen, E.C. (1998) About the asymptotic accuracy of Barron density estimates. IEEE Transactions on Information Theory 44, 999–1009. Bickel, P.J. and Sakov, A. (2002) Equality of types for the distribution of the maximum for two values of n implies extreme value type. Extremes 5, 45–53. Bingham, N.H., Goldie, C. and Teugels, J. (1987) Regular Variation. Cambridge University Press, Cambridge. Bolotin, V.A., Levy, Y. and Liu, D. (1999) Characterizing data connection and messages by mixtures of distributions on logarithmic scale. In P. Key and D. Smith (eds), Teletraffic Engineering in a Competitive World, Vol. 3b, pp. 887–896. Elsevier, Amsterdam. Bolshev, L.N. and Smirnov, N.V. (1965) Tables of Mathematical Statistics. Nauka, Moscow (in Russian). Bowman, A.W. (1982) A comparative study of some kernel-based nonparametric density estimators. Research Report No. 84/AWB11, Manchester-Sheffield School of Probability and Statistics. Bowman, A.W. (1984) An alternative method of cross-validation for the smoothing of density estimates. Biometrika 71(2), 353–360. Bratt, G. (1994) Sequential decoding for the Gilbert–Elliott channel-strategy and analysis. Doctoral thesis, Lund University. Breiman, L. (1965) On some limit theorems similar to the arc-sin law. Theory of Probability and Its Application 10, 323–331. Brockwell, P.J. and Davis, R.A. (1991) Time Series: Theory and Methods, 2nd edition. Springer, New York. Caers, J. and Van Dyck, J. (1999) Nonparametric tail estimation using a double bootstrap method. Computational Statistics & Data Analysis 29, 191–211. Capéraà, P., Fougères, A.-L. and Genest, C. (1997) A nonparametric estimation procedure for bivariate extreme value copula. Biometrika 84, 567–577. Castellana, J.V. and Leadbetter, M.R. (1986) On smoothed probability density estimation for stationary processes. Stochastic Processes and their Applications 21, 179–193. Castillo, E., Hadi, A., Balakrishnan, N. and Sarabia, J. (2006) Extreme Value and Related Models with Applications in Engineering and Science. Wiley, Hoboken, NJ. ˇ Cencov, N.N. (1982) Statistical Decision Rules and Optimal Inference. American Mathematical Society, Providence, RI. Chaudhry, M.L. (1995) On computations of the mean and variance of the number of renewals: a unified approach. Journal of the Operational Research Society 46, 1352–1364. Chen, Y., Härdle, W. and Jeong, S.-O. (2005) Nonparametric risk management with generalized hyperbolic distributions. SFB 649 Discussion Paper 2005-001, Humboldt University, Berlin. Chistyakov, V.P. (1964) A theorem on sums of independent positive random variables and its applications to branching random processes. Theory of Probability and Its Applications 9, 640–648. Chow, Y.-S., Geman, S. and Wu, L.-D. (1983) Consistent cross-validated density estimation. Annals of Statistics 11, 25–38. Coles, S. (2001) An Introduction to Statistical Modeling of Extreme Values. Springer, London. Cox, D.R. and Oakes, D. (1984) Analysis of Survival Data. Chapman & Hall, London. Crovella, M.E., Taqqu, M.S. and Bestavros, A. (1998) Heavy-tailed probability distributions in the World Wide Web. In R.J. Adler, R.E. Feldman and M.S. Taqqu (eds), A Practical Guide to Heavy Tails, pp. 3–26. Birkhäuser, Boston.

REFERENCES

297

Csörgo˝, S., Deheuvels, P. and Mason, D. (1985) Kernel estimates for the tail index of a distribution. Annals of Statistics 13, 1050–1077. Danielsson, J., de Haan, L., Peng, L. and de Vries, C. (1997) Using a bootstrap method to choose the sample fraction in tail index estimation. Technical report TI 97-016/4, Tinbergen Institute, Rotterdam. David, H.A. (1981) Order Statistics, 2nd edition. Wiley, New York. Davis, R. and Resnick, S. (1985) Limit theory for moving averages of random variables with regularly varying tail probabilities. Annals of Probability 13, 179–195. Davison, B.D. (2002) Predicting web actions from html content. In K.M. Anderson, S. Moulthrop and J. Blustein (eds), Hypertext 2002: Proceedings of the Thirteenth ACM Conference on Hypertext and Hypermedia. Association for Computing Machinery, New York. Davydov, Y., Paulauskas, V. and Raˇckauskas, A. (2000) More on P-stable convex sets in Banach spaces. Journal of Theoretical Probability 13(1), 39–64. de Haan, L. (1994) Extreme value statistics. In J. Galambos, J. Lechner, and E. Simiu (eds), Extreme Value Theory and Applications, pp. 93–122. de Haan, L. and Rootzén, H. (1993) On the estimation of high quantiles. Journal of Statistical Planning and Inference 35, 1–13. Deheuvels, P. (1973) Sur l’estimation sequentielle de la densité. Comptes Rendus de l’Académie des Sciences de Paris Série A 276, 1119–1121. Dekkers, A.L.M. and de Haan, L. (1989) On the estimation of the extreme-value index and large quantile estimation. Annals of Statistics 17(4), 1795–1832. Dekkers, A.L.M., Einmahl, J.H.J. and de Haan, L. (1989) A moment estimator for the index of an extreme value distribution. Annals of Statistics 17, 1833–1855. Deligönül, Z.S. (1985) An approximate solution of the integral equation of renewal theory. Journal of Applied Probability 22, 926–931. Devroye, L. and Györfi, L. (1985) Nonparametric Density Estimation. The L1 View. Wiley, New York. Dielman, T., Lowry, C. and Pfaffenberger, R. (1994) A comparison of quantile estimators. Communications in Statistics – Simulation and Computation 23(2), 355–371. Dietrich, D., de Haan, L. and Hüsler, J. (2002) Testing extreme value conditions. Extremes 5, 71–85. Drees, H. and Kaufmann, E. (1998) Selecting the optimal sample fraction in univariate extreme value estimation. Stochastic Processes and Their Applications 75, 149–172. Dubov, I.R. (1998) Formation of observations and approximation of the probability density function of the continuous random variable. Automation and Remote Control 59(4),281–293. Eberlein, E. and Keller, U. (1995) Hyperbolic distributions in finance. Bernoulli 1(3), 281–299. Eberlein, E. Kallsen, J. and Kristen, J. (2003) Risk management based on stochastic volatility. Journal of Risk 5, 19–44. Efron, B. and Tibshirani, R.J. (1993) An Introduction to the Bootstrap. Chapman & Hall, New York. Embrechts, P., Klüppelberg, C. and Mikosch, T. (1997) Modelling Extremal Events for Insurance and Finance. Springer, Berlin. Engl, H.W. and Gfrerer, H. (1988) A posteriori parameter choice for general regularization methods for solving linear ill-posed problems. Applied Numerical Mathematics 4, 395–417.

298

REFERENCES

Falin, G. (1990) A survey of retrial queues. Queueing Systems 7, 127–167. Falin, G. (1995) Estimation of retrial rate in a retrial queue. Queueing Systems 19, 231–246. Feinendegen, L.E., Bond, V.P., Booz, J. and Muhlensiepen, H. (1988) Biochemical and cellular mechanisms of low-dose effects. International Journal of Radiation Biology and Related Studies in Physics, Chemistry and Medicine 53(1), 23–37. Feller, W. (1968) An Introduction to Probability Theory and Its Applications I, II, 3rd edition. Wiley, New York. Feller, W. (1971) An Introduction to Probability Theory and Its Applications II, 2nd edition. Wiley, New York. Ferreira, A., de Haan, L. and Peng, L. (2000) Adaptive estimators for the endpoint and high quantiles of a probability distribution. Eurandom Research Report 99–142. Fikhtengol’ts, G.M. (1965) The Fundamentals of Mathematical Analysis. Pergamon, Oxford. Fisher, R.A. (1952) Contributions to Mathematical Statistics Wiley, New York. Fougères, A.-L. (2004) Multivariate extremes. In B. Finkenstädt and H. Rootzén (eds), Extreme Values in Finance, Telecommunications, and the Environment, pp. 373–388. Chapman & Hall, Boca Raton, FL. Franke, J., Härdle, W. and Hafner, C. (2004) Statistics of Financial Markets. Springer, Berlin. Frees, E.W. (1986a) Warranty analysis and renewal function estimation. Naval Research Logistics Quarterly 33, 361–372. Frees, E.W. (1986b) Nonparametric renewal function estimation. Annals of Statistics 14, 1366–1378. Galambos, J. (1987) The Asymptotic Theory of Extreme Order Statistics. Krieger, Malabar, FL. Gnedenko, B.V. (1943) Sur la distribution limite du terme maximum d’une série aléatoire. Annals of Mathematics 44, 423–453. Gnedenko, B.W. and Kowalenko, I.N. (1971) Einführung in die Bedienungstheorie. Oldenbourg Verlag, Munich. Goldie, C.M. and Klüppelberg, C. (1998) Subexponential distributions, In R. Adler, R. Feldman and M.S. Taqqu (eds), A Practical Guide to Heavy Tails: Statistical Techniques for Analysing Heavy Tailed Distributions, pp. 435–459. Birkhäuser, Boston. Goldie, C.M. and Smith, R.L. (1987) Slow variation with remainder: theory and applications. Quarterly Journal of Mathematics, 38, 45–71. Golub, G.H., Heath, M. and Wahba, G. (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21(2), 215–223. Gomes, M.I. and Oliveira, O. (2000) The bootstrap methodology for statistical extremes – Choice of the optimal sample fraction. Notas e Comunicações 04/2000, University of Lisbon. Grama, I. and Spokoiny, V. (2003) Pareto approximation of the tail by local exponential modeling. Preprint No. 819, Weierstrass-Institute, Berlin. Grübel, R. and Pitts, S.M. (1993) Nonparametric estimation in renewal theory 1: The empirical renewal function. Annals of Statistics 21, 1431–1451. Györfi, L., Liese, F., Vajda, I. and van der Meulen, E.C. (1998) Distribution estimates consistent in  2 -divergence. Statistics 32, 31–57. Hall, P. (1983) Large-sample optimality of least squares cross-validation in density estimation. Annals of Statistics 11, 1156–1174. Kluwer, Dordrecht. Hall, P. (1985) Asymptotic theory of minimum integrated square error for multivariate density estimation. In P.R. Krishnaiah (ed.), Proceedings of the Sixth International Symposium on Multivariate Analysis, pp. 289–309. North-Holland, Amsterdam.

REFERENCES

299

Hall, P. (1990) Using the bootstrap to estimate mean squared error and select smoothing parameter in nonparametric problems. Journal of Multivariatte Analysis 32, 177–203. Hall, P. (1992) On global properties of variable bandwidth density estimators. Annals of Statistics 20(2), 762–778. Hall, P. and Marron, J.S. (1988) Variable window width kernel estimates of probability densities. Probability Theory and Related Fields 80(1), 37–49. Hall, P. and Patil, P. (1994) On the efficiency of on-line density estimators. IEEE Transactions on Information Theory 40(5), 1504–1512. Hall, P. and Tajvidi, N. (2000) Distribution and dependence-function estimation for bivariate extreme value distributions. Bernoulli 6, 835–844. Hall, P. and Weissman, I. (1997) On the estimation of exreme tail probabilities. Annals of Statistics 25(3), 1311–1326. Hall, P. and Welsh, A.H. (1985) Adaptive estimates of parameters regular variation. Annals of Statistics 13, 331–341. Hall, P., Lahiri, S.N. and Truong, Y.K. (1995) On bandwidth choice for density estimation with dependent data. Annals of Statistics 23(6), 2241–2263. Hart, J.D. and Vieu, P. (1990) Data-driven bandwidth choice for density estimation based on dependent data. Annals of Statistics 18, 873–890. Häusler, E. and Teugels, J. (1985) On the asymptotic normality of Hill’s estimate for the exponent of regular variation. Annals of Statistics 13, 743–756. Heyman, D.P. and Sobel, M.J. (1982) Stochastic Models in Operations Research, Vol. I. McGraw-Hill, New York. Hill, B.M. (1975) A simple general approach to inference about the tail of a distribution. Annals of Statistics 3, 1163–1174. Horváth, A. and Telek, M. (2000) Approximating heavy tailed behaviour with phase type distributions. In G. Latouche and P. Taylor (eds), Advances in Algorithmic Methods for Stochastic Models, pp. 191–214. Notable Publications, Neshanic Station, NJ. Hosking, J.R.M. and Wallis, J.R. (1987) Parameter and quantile estimation for the generalized Pareto distribution. Technometrics 29, 339–349. Ivanov, V.K., Vasin, V.V. and Tanana, V.P. (1978) Theory of Linear Ill-Posed Problems and Its Applications. Nauka, Moscow (in Russian). Jureˇcková, J. and Picek, J. (2001) A class of tests on the tail index. Extremes 4, 165–183. Kettani, H. and Gubner, J.A. (2002) A novel approach to the estimation of the Hurst parameter in self-similar traffic. In Proceedings of the 27th IEEE Conference on Local Computer Networks. IEEE Computer Society, Los Alamitos, CA. Khazaeli, A.A., Tatar, M., Pletcher, S.D. and Curtsinger, J.W. (1997) Heat-induced longevity extension in Drosophila. I. Heat treatment, mortality, and thermotolerance. Journal of Gerontology: Biological Sciences 52A(1), B48–B52. Khintchine, A. and Lévy, P. (1936) Sur les lois stables. Comptes Rendus de l’Académie des Sciences de Paris 202(5), 374–376. Kilpi, J. and Lassila, P. (2006) Micro- and macroscopic analysis of RTT variability in GPRS and UMTS networks. In F. Boavida, T. Plagemann, B. Stiller, C. Westphal and E. Monteiro. (eds), Networking 2006, Lecture Notes in Computer Science 3976, pp. 1176–1181 Springer, Berlin. Kirillov, A.A. and Gvishiani, A.D. (1982) Theorems and Problems in Functional Analysis. Springer, New York. Knuth, D.E. (1973) The Art of Computer Programming. Addison Wesley, London.

300

REFERENCES

Koo, J.-Y. and Kim, W.-C. (1996) Wavelet density estimation by approximation of logdensities. Statistics and Probability Letters 26, 271–278. Koo, J.-Y. and Chung, H.-Y. (1998) Log-density estimation in linear inverse problems. Annals of Statistics 26(1), 335–362. Kooperberg, C., Stone, C.J. and Truong, Y.K. (1994) Hazard regression. Technical Report No. 389, University of California, Berkeley. Körner, U. and Nyberg, C. (1993) Load control procedures for intelligent networks. In H. Perros, G. Pujolle and Y. Takahashi (eds), Proceedings of the IFIP TC6 Task Group/WG6.4 International Workshop on Performance Communication Systems. NorthHolland, Amsterdam. Krieger, U.R., Markovitch, N.M. and Vicari, N. (2001) Analysis of World Wide Web traffic by nonparametric estimation techniques. In K. Goto, T. Hasegawa, H. Takagi, and Y. Takahashi (eds), Performance and QoS of Next Generation Networking, pp. 67–83. Springer, London. K˙us, V. and Vajda, I. (1996) A comparative study of nonparametric density estimates. Research Report 1892, Institute of Information Theory and Automation, Prague. Law, A.M. and Kelton, W.D. (2000) Simulation Modelling and Analysis, 3rd edition. McGraw-Hill, New York. Leadbetter, M.R. (1983) Extremes and local dependence in stationary sequences. Probability Theory and Related Fields 65(2), 291–306. Leadbetter, M.R., Lingren, G. and Rootzén, H. (1983) Extremes and Related Properties of Random Sequences and Processes. Springer, Berlin. Lévy, P. (1925) Calcul des probabilités. Paris: Gauthier Villars. Luckey, T.D. (1980) Hormesis with Ionizing Radiation. CRC Press, Boca Raton, FL. Maiboroda, R.E. and Markovich, N.M. (2004) Estimation of heavy-tailed probability density function with application to Web data. Computational Statistics 19(4), 569–592. Markovich, N.M. (1989) Experimental analysis of nonparametric probability density estimates and of methods for smoothing them. Automation and Remote Control 50, 941–948. Markovich, N.M. (1995) Mathematical concepts, In W. Morgenstern, V.K. Ivanov, A.I. Michalski, A.F. Tsyb and G. Schettler (eds), Mathematical Modelling with Chernobyl Registry Data. Springer, Berlin. Markovich, N.M. (1998) Regularization of some linear integral equations of population analysis. Automation and Remote Control 59(3), 418–431. Markovich, N.M. (2000) Detection of hormesis by empirical data as an ill-posed problem. Automation and Remote Control 61(1), Part 2, 133–143. Markovich, N.M. (2002) Transformed estimates of densities of heavy-tailed distributions and classification. Automation and Remote Control 63(4), 627–640. Markovich, N.M. (2004) Nonparametric renewal function estimation and smoothing by empirical data. Technical Report, ETH, Zurich. Markovich, N.M. (2005a) On-line estimation of the tail index for heavy-tailed distributions with application to WWW-traffic. In Proceedings of 1st Conference on Next Generation Internet Design and Engineering, pp. 388–395. IEEE, Piscataway, NJ. Markovich, N.M. (2005b) High quantile estimation for heavy-tailed distributions. Performance Evaluation 62(1–4), 178–192. Markovich, N.M. (2005c) Accuracy of transformed kernel density estimates for a heavy-tailed distribution. Automation and Remote Control 66(2), 217–232.

REFERENCES

301

Markovich, N.M. (2006a) Estimation of heavy-tailed density functions with application to WWW-traffic. In Proceedings of 2nd Conference on Next Generation Internet Design and Engineering, pp. 208–215. IEEE, Piscataway, NJ. Markovich, N.M. (2006b) Estimation of marginal density by dependent data. In Proceedings, Stochastic Performance Models for Resource Allocation in Communication Systems, Amsterdam, 8–10 November. http://www.cwi.nl/events/2006/StoPeRa. Markovich, N.M. (2008) Load control by arrivals of TCP connections. In Proceedings of VII International Conference, System Identification and Control Problems, Moscow, January 28–31, submitted. Markovich, N.M. and Kilpi, J. (2006) Bivariate statistical analysis of TCP-flow sizes and durations. In Proceedings, Stochastic Performance Models for Resource Allocation in Communication Systems, Amsterdam, 8–10 November. http://www. cwi.nl/events/2006/StoPeRa/. Markovitch, N.M. and Krieger, U.R. (1999) Estimating basic characteristics of arrival processes in advanced packet-switched networks by empirical data. Proceedings of the First IEEE/Popov Workshop on Internet Technologies and Services, pp. 70–78. IEEE, New York. Markovitch, N.M. and Krieger, U.R. (2000) Nonparametric estimation of long-tailed density functions and its application to the analysis of World Wide Web traffic. Performance Evaluation 42(2–3), 205–222. Markovitch, N.M. and Krieger, U.R. (2002a) The estimation of heavy-tailed probability density functions, their mixtures and quantiles. Computer Networks 40(3), 459–474. Markovitch, N.M. and Krieger, U.R. (2002b) Estimating basic characteristics of arrival processes in telecommunication networks by empirical data. Telecommunication Systems 20, 11–31. Markovich, N.M. and Krieger, U.R. (2006a) Nonparametric estimation of the renewal function by empirical data. Stochastic Models 22(2), 175–199. www.informaworld.com. Markovich, N.M. and Krieger, U.R. (2006b) Statistical inspection and analysis techniques for traffic data arising from the Internet. In Proceedings of the HET-NETs ’04 2nd International Working Conference on Performance Modelling and Evaluation of Heterogeneous Networks, July 26–28, Ilkley, West Yorkshire, pp. 72/1–72/9. Markovich, N.M. and Michalski, A.I. (1995) Estimation of health indicates from data on revealed morbidity. Automation and Remote Control 56(7), 1033–1041. Markovich, N.M., Morgenstern, W. and Michalski, A.I. (1996) Semi-Markov identification based on the small samples approach. In Proceedings of the 10th European Simulation Multiconference, pp. 791–795. Society for Computer Simulation International. Martynov, G.V. (1978) Omega-Square Criteria. Nauka, Moscow (in Russian). Mason, D.M. (1982) Laws of large numbers for sums of extreme values. Annals Probability 10, 754–764. Mason, D.M. and Turova, T.S. (1994) Weak convergence of the Hill estimator process. In J. Galambos, J. Lechner and E. Simiu (eds), Extreme Value Theory and Applications, pp. 419–431. Kluwer, Dordrecht. Matthys, G. and Beirlant, J. (2001) Estimating the extreme value index and high quantiles with exponential regression models. Preprint, Center for Statistics, University of Leuven. Matthys, G. and Beirlant, J. (2003) Estimating the extreme value index and high quantiles with exponential regression models. Statistica Sinica 13, 853–880.

302

REFERENCES

Matthys, G., Delafosse, E., Guillou, A. and Beirlant, J. (2004) Estimating catastrophic quantile levels for heavy-tailed distributions. Insurance: Mathematics and Economics 34, 517–537. McConalogue, D.J. (1981) Numerical treatment of convolution integrals involving distributions with densities having singularities at the origin. Communications in Statistics, Series B10, 265–280. McNeil, A.J. and Saladin, T. (1997) The peaks over thresholds method for estimating high quantiles of loss distributions. In Proceedings of 28th International ASTIN Colloquium. McNeil, A.J., Frey, R. and Embrechts, P. (2005) Quantitative Risk Management: Concepts, Techniques, Tools. Princeton University Press, Princeton, NJ. Michalski, A.I. (1987) Choosing an algorithm of estimation based on samples of limited size. Automation and Remote Control 48(7), 909–918. Mikosch, T. (1999) Regular variation, subexponentiality and their applications in probability theory. Technical Report 99-013, University of Groningen. Mikosch, T. (2004) Modeling dependence and tails of financial time series. In B. Finkenstädt and H Rootzén (eds), Extreme Values in Finance, Telecommunications, and the Environment, pp. 187–286. Chapman & Hall, Boq Raton, FL. Mikosch, T. (2006) Copulas: Tales and facts. Extremes 9, 3–20. Mikosch, T. and Nagaev, A.V. (1998) Large deviations for heavy-tailed sums with applications to insurance. Extremes 1, 81–110. Mohan, N.R. (1976) Teugels’ renewal theorem and stable laws. Annals of Probability 4(5), 863–868. Morozov, V.A. (1984) Methods for Solving Incorrectly Posed Problems. Springer, New York. Murthy, V.K. (1966) Nonparametric estimation of multivariate densitites with applications. In P.R. Krishnaiah (ed.), Multivariate Analysis, pp. 43-48. Academic, New York. Nabe, M., Murata, M. and Miyahara, H. (1998) Analysis and modeling of World Wide Web traffic for capacity dimensioning of Internet access lines. Performance Evaluation 34, 249–271. Nadaraya, E.A. (1965) About the nonparametrical estimates of the probability density and regression. Probability Theory and Its Applications 10(1), 199–203. Naito, K. (2001) On a certain class of nonparametric density estimators with reduced bias. Statistics and Probability Letters, 51, 71–78. Nelsen, R.B. (1998) An Introduction to Copulas. Springer, New York. Newell, G.F. (1964) Asymptotic extremes for m-dependent random variables. Annals of Mathematical Statistics 35, 1322–1325. Novak, S.Y. (1996) On the distribution of the ratio of sums of random variables. Theory of Probability and Its Applications 41(3), 479–503. Novak, S.Y. (1999) Generalised kernel density estimator. Theory of Probability and Its Applications, 44(3): 570–583. Novak, S.Y. (2002) Inference of heavy tails from dependent data. Siberian Advances in Mathematics 12(2), 73–96. Ohta, H. and Kitani, T. (1990) Simulation study of the cell discard process and the effect of cell loss compensation in ATM networks. IEICE Transactions E 73, (10), 1704–1711. Padmanabhan, V. and Mogul, J.C. (1996) Using preditive prefetching to improve World Wide Web latency. ACM SIGCOMM Computer Communication Review 26(3). Park, B.U. and Marron, J.S. (1990) Comparison of data-driven bandwidth selectors. Journal of the American Statistical Association 85, 66–72.

REFERENCES

303

Parzen, E. (1962) On estimation of a probability density function and mode. Annals of Mathematical Statistics 33(3), 1065–1076. Paulauskas, V. (2003) A new estimator for tail index. Acta Applicandae Mathematica, 79(1/2), 167–175. Petrov, V.V. (1975) Sums of Independent Random Variables. Springer, New York. Pickands, J. (1975) Statistical inference using extreme order statistics. Annals of Statistics 3, 119–131. Pickands, J. (1981) Multivariate extreme value distributions. In Bulletin of the International Statistical Institute, Proceedings of the 43rd Session (Buenos Aires), pp. 859–878. Voorburg: 151. Planel, H., Soleilhavoup, J.P., Giess, M.C. and Tixador, R. (1967) Demonstration of a retardation of development of Drosophila melanogaster by diminution of environmental natural radioactivity. Comptes Rendus Hebdomadaires des Séances de l’Académie des Sciences, Serie D: Sciences Naturelles 264(6), 865–868 (in French). Polzehl, J. and Spokoiny, V. (2002) Local likelihood modeling by adaptive weights smoothing. Preprint, No. 787, Weierstrass-Institute, Berlin. Prakasa Rao, B.L.S. (1983) Nonparametric Functional Estimation. Academic, Orlando, FL. Prudnikov, A.P., Brychkov Yu.A. and Marichev, O.I. (1981) Integrals and Expansions. Elementary Functions. Nauka, Moscow (in Russian). Reiss, R.D. (1975) Consistency of a certain class of empirical density functions. Metrika 22(4), 189–203. Reiss, R.D. (1989) Approximate Distributions of Order Statistics. Springer, New York. Reiss, R.D. and Thomas, M. (2005) Statistical Analysis of Extreme Values (for Insurance, Finance, Hydrology and Other Fields), 3rd revised edition. Birkhäuser, Basel. Rényi, A. (1953) On the theory of order statistics. Acta Mathematicae Academiae Scientiarum Hungaricae 4, 191–232. Resnick, S.I. (1997) Heavy tail modeling and teletraffic data. With discussion and a rejoinder by the author. Annals of Statistics 25, 1805–1869. Resnick, S.I. (2006) Heavy-Tail Phenomena. Probabilistic and Statistical Modeling. Springer, New York. Resnick, S.I. and Stˇaricˇa, C. (1999) Smoothing the moment estimate of the extreme value parameter. Extremes 1(3), 263–294. Roppel, C. (1999) Estimating cell transfer delay and cell delay variation in ATM networks: Measurement techniques and results. European Transactions on Telecommunications 10(1), 13–21. Rosenblatt, M. (1956a) Remarks on some nonparametric estimates of a density function. Annals of Mathematical Statistics, 27(3), 832–837. Rosenblatt, M. (1956b) A central limit theorem and a strong mixing condition. Proceedings of the National Academy of Sciences of the United States of America 42, 43–47. Rudemo, M. (1982) Empirical choice of histogram and kernel density estimators. Scandinavian Journal of Statistics 9, 65–78. Sachs, R.K., Hlatky, L., Hahnfeldt, P. and Chen, P.L. (1990) Incorporating doserate effects in Markov radiation cell-survival models. Radiation Research 124(2), 216–226. Sagan, L.A. (1987) What is hormesis and why haven’t we heard about it before? Health Physics 52(5), 521–525. Schneider, H., Lin, B.-S. and O’Cinneide, C. (1990) Comparison of nonparametric estimators for the renewal function. Applied Statistics 39(1), 55–61.

304

REFERENCES

Schuster, E.F. and Gregory, G.G. (1981) On the nonconsistency of maximum likelihood nonparametric density estimators. In W.F. Eddy (ed.) Computer Science and Statistics: Proceedings of 13th Symposium on the Interface, pp. 295–298. Springer, New York. Schuster, J. (1985) Incorporating support constraints into nonparametric estimators of densities. Communications in Statistics – Theory and Methods 14(5), 1123–1136. Scott, D.W. (1992) Multivariate Density Estimation Theory, Practice and Visualization. Wiley, New York. Sgibnev, M.S. (1981) Renewal theorem in the case of an infinite variance. Siberian Mathematical Journal 22, 787–796. Shao, J. and Tu, D. (1995) The Jackknife and Bootstrap, Springer, New York. Sigman, K. (1999) Appendix: A primer on heavy-tailed distributions. Queueing Systems 33, 261–275. Silverman, B.W. (1986) Density Estimation for Statistics and Data Analysis. Chapman & Hall, New York. Simonoff, J.S. (1996) Smoothing Methods in Statistics. Springer, New York. Smirnov, N.V. and Dunin-Barkovsky, I.V. (1965) Course in Probability Theory and Mathematical Statistics for Technical Applications. Nauka, Moscow (in Russian). Smith, R.L. (1987) Estimating tails of probability distributions. Annals of Statistics 15, 1174–1207. Smith, W.L. (1954) Asymptotic renewal theorems. Proceedings Section A: Mathematics – Royal Society of Edinburgh 64, 9–48. Stefanyuk, A.R. (1980) Convergence rate of a class of probability density estimates. Automation and Remote Control 40, 1706–1711. Stefanyuk, A.R. (1984) Estimation of the probability density function. In V.N. Vapnik (ed.), Algorithms and Programs for Dependency Reconstruction pp. 688–706. Nauka, Moscow (in Russian). Stefanyuk, A.R. (1986) Estimating the likelihood ratio function in the problem of ‘failure’ of a stochastic process. Automation and Remote Control 47(9), 1210–1216. Stefanyuk, A.R. (1992) The problem of nonparametric estimation of mortality risk function. In A.B. Kurzhanski and V.M. Veliov (eds), Modeling Techniques for Uncertain Systems, pp. 53–67. Birkhäser, Boston. Stefanyuk, A.R. and Karandeev, D.A. (1996) Parameter choice of adaptation algorithm of density estimation by empirical data. Automation and Remote Control 57(10), 1453–1466. Stone, C.J. (1984) An asymptotically optimal window selection rule for kernel density estimates. Annals of Statistics 12, 1285–1297. Stone, M. (1974) Cross-validation choice and assesment of statistical predictions (with discussion). Journal of Royal Statistical Society B 36, 111–147. Stratonovich, R.L. (1969) The rate of convergence of algorithms of the probability density estimation. Izvestiya Academii Nauk USSR, Technical Cybernetics Series 6(1), 3–15 (in Russian). Tarasenko, F.P. (1968) On the evaluation of an unknown probability density function, the direct estimation of the entropy from independent observations of a continuous random variable and the distribution-free test of goodness-of-fit. Proceedings of the IEEE 56(1), 2052–2053. Tarasenko, F.P. (1976) Neparametricheskaya statistika (Nonparametric Statistics). Tomsk State University Press, Tomsk (in Russian). Teugels, J.L. (1968) Renewal theorems when the first or the second moment is infinite Annals of Statistics 39, 1210–1219.

REFERENCES

305

Tikhonov, A.N. and Arsenin, V.Y. (1977) Solution of Ill-Posed Problems. Wiley, New York. Trivedi, K.S. (1997) Probability & Statistics with Reliability, Queuing, and Computer Science Applications. Prentice Hall of India, New Delhi. Vajda, I. and van der Meulen, E.C. (2001) Optimization of Barron density estimates. IEEE Transactions on Information Theory 47(5), 1867–1883. Vapnik, V.N. (1982) Estimation of Dependences Based on Empirical Data. Springer, New York. Vapnik, V.N. (1984) Algorithms and Programmes for Dependency Reconstruction. Nauka, Moscow (in Russian). Vapnik, V.N. and Stefanyuk, A.R. (1979) Nonparametric methods for probability density reconstruction. Automation and Remote Control 39, 1127–1140. Vapnik, V.N., Markovich, N.M. and Stephanyuk, A.R. (1992) Rate of convergence in L2 of the projection estimator of the distribution density, Automation and Remote Control 53, 677–686. Vaupel, J.W., Manton, K.G. and Stallard, E. (1979) The impact of heterogeneity in individual frailty on the dynamics of mortality. Demography 16, 439–454. Vicari, N. (1997) Measurement and modelling of WWW-sessions. Technical Report No. 184, Institute of Computer Science, University of Würzburg. Wahba, G. (1981) Data-based optimal smoothing of orthogonal series density estimates. Annals of Statistics 9, 146–156. Wand, M.P. and Jones, M.C. (1995) Kernel Smoothing. Chapman & Hall, London. Wand, M.P., Marron, J.S. and Ruppert, D. (1991) Transformations in density estimation, Journal of American Statistical Association, Theory and Methods 86, 343–353. Weissman, I. (1978) Estimation of parameters and large quantiles based on the k largest observations. Journal of American Statistical Association 73, 812–815. Weissman, I. (2005) Two dependence measures for multivariate extreme value distributions. Preprint. Wegman, E.J. and Davies, H.I. (1979) Remarks on some recursive estimators of a probability density. Annals of Statistics 7(2), 316–327. Wertz, W. (1985) Sequential and recursive estimators of the probability density. Statistics 16, 277–295. Willinger, W., Taqqu, M.S., Leland, W.E. and Wilson, D.V. (1995) Self-similarity in highspeed packet traffic: Analysis and modeling of ethernet traffic measurements. Statistical Science 10(1), 67–85. Wolverton, C.T. and Wagner, T.J. (1969) Asymptotically optimal discriminant functions for pattern classification. IEEE Transactions on Information Theory IT-15, 258–265. Xie, M. (1989) On the solution of renewal-type integral equations. Communications in Statistics – Simulation and Computation 18(1), 281–293. Yakovlev, A.Y., Tsodikov, A.D. and Bass, L. (1993) A stochastic model of hormesis. Mathematical Biosciences 116, 197–219. Yamato, H. (1971) Sequentional estimation of a continuous probability density function and mode Bulletin of Mathematical Statistics 14, 1–12. Yang, L. and Marron, J.S. (1999) Iterated transformation-kernel density estimation. Journal of the American Statistical Association 94, 580–589. Yashin, A.I., Andreev, K.F., Khazaeli, A., Curtsinger, J.W. and Vaupel, J.W. (1996) Death-after-stress-data in the analysis of heterogeneous mortality. In G. Kristensen (ed.), Symposium i Anvendt Statistik, Odense University, January 22–24, pp. 24–36.

Index algorithm Bayesian classification 117 for boundary kernel selection 139 of density estimation based on adaptive transformation 128 of density estimation based on fixed transformation 125 of double bootstrap 12 of sequential procedure 12 asymptotical mean integrated squared error 115 bandwidth of kernel 70 Bartlett’s formula 44 Bayesian risk of misclassification 155 bin width of histogram 76 BISDN 208 bivariate quantile curve 55 bootstrap classical 10, 369 estimate of renewal function 228 non-classical 369 re-sample 9, 229 boundary effect of kernel estimates 73 censoring 94 characteristic number 192 classifier 152 Bayesian 152 empirical Bayesian 152 component-wise maxima 50

condition Cramér’s 5, 225 Hall’s 8 Hölder (or Lipschitz) 68, 190 mixing 42 von Mises 180 confidence interval of group estimate 18 of high quantile 172 of renewal function 226, 227, 228 consistency strong 73 weak 73 convex hull 50 copula 33, 49 cross-validation 74, 77 for dependent data 91 integrated squared error 78, 115 weighted integrated squared error 115 density estimation approach L1 62 L2 63  2 63 dependence index sequence 86 long range 45, 86, 89 short range 45, 89 distribution bivariate extreme value 49 exponential 234

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

308

INDEX

distribution (Continued ) extreme value xiii fitted 119, 129 Fréchet 23 function 2 gamma 234 generalized extreme value 3, 54 generalized hyperbolic 92 generalized Paret xiii, 14, 129 heavy-tailed 3 interarrival-time 220 isosceles triangular 118 light-tailed 3 normal inverse Gaussian 92 Pareto 239, 241 Poisson 208 regularly varying 4, 168, 227 stable 18 subexponential 3 target 119, 129 Weibull 172, 188, 234 dose–effect dependence 204 eigenfunction 192 equation Fredholm’s 66 Volterra’s 185, 198, 211 estimate maximum likelihood 167 re-transformed 118 estimator Barron 112 based on exponential regression model 14 combined parametric-nonparametric 101, 165 EVI kernel 13 Frees’ 223, 233 group 19 Hill’s 6 histogram-type of renewal function 224 intensity of nonhomogeneous Poisson process 209 Kaplan–Meier 206 modified Weissman’s 166 moment 13 on-line 20

Parzen–Rosenblatt kernel 68, 70 Pickands’ 13 polygram 76, 126, 131 POT 14, 165 projection 63, 74 ratio 7 smoothed projection 69 UH 13 variable bandwidth kernel 113 weighted quantile 164 Weissman’s 165 Euler’s constant 231 expected shortfall 92 failure time 181, 199 Fourier coefficients 193 frailty 201, 202 function autocorrelation 43 auto-covariance 45 covariance 42 empirical distribution 67 empirical mean excess 8, 28 hazard rate 180 Laplace’s 25 leave-out-l cross-validation 91 mean excess 28 moment generating 5, 225 Pickands dependence 49 ratio of the hazard rates 181, 198 renewal 220, 221 sample autocorrelation 43 sample heavy-tailed autocorrelation 43 slowly varying 4 survival 94, 181 functional regularization 68 stabilizing 67 high quantile 163 Hill plot 39 hormesis 200 independent random variables 2 index extreme value 3 tail 3

INDEX inequality Hoeffding’s 260, 262, 266 Hölder 269 intensity of nonhomogeneous Poisson process 208 inter-arrival times 221 kernel boundary 132 Epanechnikov’s 71 Gaussian 71 modified bi-weight 136 triangular 132 Kullback’s metric 62 Laplace–Stieltjes transform 221 leave-out sequence 91 lemma Borel–Cantelli 256, 258, 261, 266, 286, 287, 289 of inverse operator 183 lifetime 181 likelihood ratio 198 maximum domain of attraction 3 mean integrated squared error 135 mean risk 104 mean squared error 9, 72, 88 method block maxima xiii D 82, 127 discrepancy 80, 115, 119, 127 elemental percentile 15, 167 exponent 64 Lagrange 183 least-squares 81 maximum likelihood 53, 62, 167 of mismatch 184 of moments 15 POT xiii, 176 of probability-weighted moments 15, 54, 167 regularization 67, 183 structural risk-minimization 104 Xie’s RS 234 2 82, 127, 147 model Gilbert–Elliott Markov 210

309

Pareto-type 165 retrial 213 semi-Markov 182, 193 Monte-Carlo study 23 mortality risk 94, 180 table 94 operator adjoint 184 self-adjoint (Hermitian) 192 operator equation 182 orbit 212 order of kernel 70 order statistics 6 orthonormalized system 192 outliers xii, 117 over-smoothing bandwidth selector

77

parameter Hurst 45 smoothing 76 Parseval equality 277, 278 pattern recognition 151 plot of histogram-type estimate 232 probability density function 2 of misclassification 152 space 1, 182 problem correct by Hadamard 67 ill-posed 67, 182 inverse 181 process ARMA 43 exactly second-order self similar 47 GARCH 44 log-return 91 MA 45 second order stationary 45 QQ plot 28, 165 quantile 163 random variable 2 regularization parameter 67, 183

310

INDEX

representation Jenkinson–von Mises 3 Karamata of slowly varying function 227 Rényi 272 retrial call definition A 213 definition B 213 retrial queues 212 right endpoint of distribution 5, 28 scheme Fisher’s 62 transform–retransform 117, 128 second-order asymptotic relation 18 selection of k in Hill’s estimate bootstrap 9 double bootstrap 12 exceedance plot 8 Hill’s plot 8 sequential procedure 12 space Hölder 190 Sobolev 65 statistic Kolmogorov–Smirnov 80 Mises–Smirnov 80, 148

Rényi 191 Stieltjes convolution 221 Taylor’s expansion 255 TCP-flow 55 theorem Glivenko–Cantelli 198, 211 Lebesgue’s 61 Pickands’ 5 Scheffé’s 63 Sklar’s 33, 49 Smith’s 221 total variation 63 transformation adaptive 128 fixed 118, 124 function 117 transmission control protocol 30 u-statistic 223 value at risk 92 Vapnik–Chervonenkis dimension 104 warranty control 221 wavelet basis 75 Web prefetching 161 Web traffic 30

WILEY SERIES IN PROBABILITY AND STATISTICS established by Walter A. Shewhart and Samuel S. Wilks Editors David J. Balding, Pater Bloomfield, Noel A. C. Cressie, Nicholas I. Fisher, Iain M. Johnstone, J. B. Kadane, Geert Molenberghs, Louise M. Ryan, David W. Scott, Adrian F. M. Smith Editors Emeriti Vic Barnett, J. Stuart Hunter, David G. Kendall, Jozef L. Teugels The Wiley Series in Probability and Statistics is well established and authoritative. It covers many topics of current research interest in both pure and applied statistics and probability theory. Written by leading statisticians and institutions, the titles span both state-of-the-art developments in the field and classical methods. Reflecting the wide range of current research in statistics, the series encompasses applied, methodological and theoretical statistics, ranging from applications and new techniques made possible by advances in computerized practice to rigorous treatment of theoretical approaches. This series provides essential and invaluable reading for all statisticians, whether in academia, industry, government, or research. ABRAHAM and LEDOLTER · Statistical Methods for Forecasting AGRESTI · Analysis of Ordinal Categorical Data AGRESTI · An Introduction to Categorical Data Analysis AGRESTI · Categorical Data Analysis, Second Edition ALTMAN, GILL, and McDONALD · Numerical Issues in Statistical Computing for the Social Scientist AMARATUNGA and CABRERA · Exploration and Analysis of DNA Microarray and Protien Array Data ANDÉL · Mathematics of Chance ANDERSON · An Introduction to Multivariate Statistical Analysis, Third Edition ∗ ANDERSON · The statistical Analysis of Time Series ANDERSON, AUQUIER, HAUCK, OAKES, VANDAELE, and WEISBERG · Statistical Methods for Comparative Studies ANDERSON and LOYNES · The Teaching of Practical Statistics ARMITAGE and DAVID · (EDITORS) Advances in Biometry ARNOLD, BALAKRISHNAN, and NAGARAJA · Records ∗ ARTHANARI and DODGE · Mathematical Programming in Statistics ∗ BAILEY · The Elements of Stochastic Processes with Applications to the Natural Sciences BALAKRISHNAN and KOUTRAS · Runs and Scans with Applications BALAKRISHNAN AND NG · Precedence-Type Tests and Applications BARNETT · Comparative Statistical Inference, Third Edition BARNETT · Environmental Statistics: Methods & Applications BARNETT and LEWIS · Outliers in Statistical Data, Third Edition BARTOSZYNSKI and NIEWIADOMSKA-BUGAJ · Probability and Statistical Inference BASILEVSKY · Statistical Factor Analysis and Related Methods: Theory and Applications BASU and RIGDON · Statistical Methods for the Reliability of Repairable Systems BATES and WATTS · Nonlinear Regression Analysis and Its Applications BECHHOFER, SANTNER, and GOLDSMAN · Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons BELSLEY · Conditioning Diagnostics: Collinearity and Weak Data in Regression BELSLEY, KUH, and WELSCH · Regression Diagnostics: Identifying Influential Data and Sources of Collinearity BENDAT and PIERSOL · Random Data: Analysis and Measurement Procedures, Third Edition BERNARDO and SMITH · Bayesian Theory BERRY, CHALONER, and GEWEKE · Bayesian Analysis in Statistics and Econometrics: Essays in Honor of Arnold Zellner BHAT and MILLER · Elements of Applied Stochastic Processes, Third Edition BHATTACHARYA and JOHNSON · statistical Concepts and Methods

∗

Now available in a lower priced paperback edition in the Wiley Classics Library.

Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice © 2007 John Wiley & Sons, Ltd. ISBN: 978-0-470-51087-2

N. Markovich

BHATTACHARYA and WAYMIRE · Stochastic Processes with Applications BIEMER, GROVES, LYBERG, MATHIOWETZ, and SUDMAN · Measurement Errors in Surveys BILLINGSLEY · Convergence of Probability Measures, Second Edition BILLINGSLEY · Probability and Measure, Third Edition BIRKES and DODGE · Alternative Methods of Regression BLISCHKE and MURTHY · (editors) Case Studies in Reliability and Maintenance BLISCHKE and MURTHY Reliability: Modeling, Prediction, and Optimization BLOOMFIELD · Fourier Analysis of Time Series: An Introduction, Second Edition BOLLEN · Structural Equations with Latent Variables BOLLEN and CURRAN · Latent Curve Models: A Structural Equation Perspective BOROVKOV · Ergodicity and Stability of Stochastic Processes BOSQ and BLANKE · inference and Prediction in Large Dimensions BOULEAU · Numerical Methods for Stochastic Processes BOX · Bayesian inference in Statistical Analysis BOX R. A. Fisher, · the Life of a Scientist BOX and DRAPER · Empirical Model–Building and Response Surfaces ∗ BOX and DRAPER · Evolutionary Operation: A Statistical Method for Process Improvement BOX, HUNTER, and HUNTER · Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building BOX, HUNTER, and HUNTER · Statistics for Experimenters: Design, Innovation and Discovery, Second Edition BOX and LUCE NO · Statistical Control by Monitoring and Feedback Adjustment BRANDIMARTE · Numerical Methods in Finance: A MATLAB–Based Introduction BROWN and HOLLANDER · Statistics: A Biomedical Introduction BRUNNER, DOMHOF, and LANGER · Nonparametric Analysis of Longitudinal Data in Factorial Experiments BUCKLEW · Large Deviation Techniques in Decision, Simulation, and Estimation CAIROLI and DALANG · Sequential Stochastic Optimization CASTILLO, HADI, BALAKRISHNAN and SARABIA · Extreme Value and Related Models with Applications in Engineering and Science CHAN · Time Series: Applications to Finance CHATTERJEE AND HADI · Regression Analysis by Example, Fourth Edition CHATTERJEE and HADI · Sensitivity Analysis in Linear Regression CHATTERJEE and PRICE · Regression Analysis by Example, Third Edition CHERNICK · Bootstrap Methods: A Practitioner’s Guide CHERNICK and FRIIS · introductory Biostatistics for the Health Sciences CHILÉS and DELFINER · Geostatistics: Modeling Spatial Uncertainty CHOW and LIU Design and Analysis of Clinical Trials: Concepts and Methodologies, Second Edition CLARKE and DISNEY · Probability and Random Processes: A First Course with Applications, Second Edition ∗ COCHRAN and COX · Experimental Designs, Second Edition CONGDON · Applied Bayesian Modelling CONGDON · Bayesian Models for Categorical Data CONGDON · Bayesian Statistical Modelling CONGDON · Bayesian Statistical Modelling, Second Edition CONOVER · Practical Nonparametric Statistics, Second Edition COOK · Regression Graphics COOK and WEISBERG · An introduction to Regression Graphics COOK and WEISBERG · Applied Regression Including Computing and Graphics CORNELL · Experiments with Mixtures, Designs, Models, and the Analysis of Mixture data, Third Edition COVER and THOMAS · Elements of Information Theory COX · A Handbook of Introductory Statistical Methods ∗ COX · Planning of Experiments CRESSIE · Statistics for Spatial Data, Revised Edition ∗

Now available in a lower priced paperback edition in the Wiley Classics Library.

CSÖRGÖ and HORVÁTH · Limit Theorems in Change Point Analysis DANIEL · Applications of Statistics to Industrial Experimentation DANIEL · Biostatisties: A Foundation for Analysis in the Health Sciences, Sixth Edition ∗ DANIEL · Fitting Equations to Data: Computer Analysis of Multifactor Data, Second Edition DASU and JOHNSON · Exploratory Data Mining and Data Cleaning DAVID and NAGARAJA · Order Statistics, Third Edition ∗ DEGROOT, FIENBERG, and KADANE · Statistics and the Law DEL CASTILLO · Statistical Process Adjustment for Quality Control DEMARIS · Regression with Social Data: Modeling Continuous and Limited Response Variables DEMIDENKO · Mixed Models: Theory and Applications DENISON, HOLMES, MALLICK, and SMITH· Bayesian Methods for Nonlinear Classification and Regression DETTE and STUDDEN · The Theory of Canonical Moments with Applications in Statistics, Probability, and Analysis DEY and MUKERJEE · Fractional Factorial Plans DILLON and GOLDSTEIN · Multivariate Analysis: Methods and Applications DODGE · Alternative Methods of Regression ∗ DODGE and ROMIG · Sampling Inspection Tables, Second Edition ∗ DOOB · Stochastic Processes DOWDY, WEARDEN, and CHILKO · Statistics for Research, Third Edition DRAPER and SMITH · Applied Regression Analysis, Third Edition DRYDEN and MARDIA · Statistical Shape Analysis DUDEWICZ and MISHRA · Modern Mathematical Statistics DUNN and CLARK · Applied Statistics: Analysis of Variance and Regression, Second Edition DUNN and CLARK · Basic Statistics: A Primer for the Biomedical Sciences, Third Edition DUPUIS and ELLIS · A Weak Convergence Approach to the Theory of Large Deviations EDLER and KITSOS · (editors) Recent Advances in Quantitative Methods in Cancer and Human Health Risk Assessment ∗ ELANDT-JOHNSON and JOHNSON · Survival Models and Data Analysis ENDERS · Applied Econometric Time Series ETHIER and KURTZ Markov · Processes: Characterization and Convergence EVANS, HASTINGS, and PEACOCK · Statistical Distribution, Third Edition FELLER · An introduction to Probability Theory and Its Applications, Volume I, Third Edition, Revised; Volume II, Second Edition FISHER and VAN BELLE · Biostatistics: A Methodology for the Health Sciences FITZMAURICE, LAIRD, and WARE · Applied Longitudinal Analysis ∗ FLEISS · The Design and Analysis of Clinical Experiments FLEISS · Statistical Methods for Rates and Proportions, Second Edition FLEMING and HARRINGTON · Counting Processes and Survival Analysis FULLER · Introduction to Statistical Time Series, Second Edition FULLER · Measurement Error Models GALLANT · Nonlinear Statistical Models GEISSER · Modes of Parametric Statistical Inference GELMAN and MENG · (editors) Applied Bayesian Modeling and Casual Inference from Incomplete-data Perspectives GEWEKE · Contemporary Bayesian Econometrics and Statistics GHOSH, MUKHOPADHYAY, and SEN · Sequential Estimation GIESBRECHT and GUMPERTZ · Planning, Construction, and Statistical Analysis of Comparative Experiments GIFI · Nonlinear Multivariate Analysis GIVENS and HOETING · Computational Statistics GLASSERMAN and YAO · Monotone Structure in Discrete-Event Systems GNANADESIKAN · Methods for Statistical Data Analysis of Multivariate Observations, Second Edition GOLDSTEIN and LEWIS · Assessment: Problems, Development, and Statistical Issues GREENWOOD and NIKULIN · A Guide to Chi-Squared Testing ∗

Now available in a lower priced paperback edition in the Wiley Classics Library

GROSS and HARRIS · Fundamentals of Queueing Theory, Third Edition HAHN and SHAPIRO · Statistical Models in Engineering HAHN and MEEKER · Statistical Intervals: A Guide for Practitioners HALD · A History of Probability and Statistics and their Applications Before 1750 HALD · A History of Mathematical Statistics from 1750 to 1930 HAMPEL · Robust Statistics: The Approach Based on Influence Functions HANNAN and DEISTLER · The Statistical Theory of Linear Systems HEIBERGER · Computation for the Analysis of Designed Experiments HEDAYAT and SINHA · Design and Inference in Finite Population Sampling HEDEKER and GIBBONS · Longitudinal Data Analysis HELLER MACSYMA · for Statisticians HINKELMANN and KEMPTHORNE · Design and Analysis of Experiments, Volume 1: Introduction to Experimental Design HINKELMANN and KEMPTHORN · Design and Analysis of Experiments, Volume 2: Advanced Experimental Design HOAGLIN, MOSTELLER, and TUKEY · Exploratory Approach to Analysis of Variance HOAGLIN, MOSTELLER, and TUKEY · Exploring Data Tables, Trends and Shapes ∗ HOAGLIN, MOSTELLER, and TUKEY · Understanding Robust and Exploratory Data Analysis HOCHBERG and TAMHANE · Multiple Comparison Procedures HOCKING · Methods and Applications of Linear Models: Regression and the Analysis of Variance, Second Edition HOEL · Introduction to Mathematical Statistics, Fifth Edition HOGG and KLUGMAN LOSS · Distributions HOLLANDER and WOLFE · Nonparametric Statistical Methods, Second Edition HOSMER and LEMESHOW · Applied Logistic Regression, Second Edition HOSMER and LEMESHOW · Applied Survival Analysis: Regression Modeling of Time to Event Data HUBER · Robust Statistics HUBERTY · Applied Discriminant Analysis HUNT and KENNEDY · Financial Derivatives in Theory and Practice, Revised Edition HUSKOVA, BERAN, and DUPAC · Collected Works of Jaroslav Hajek—with Commentary HUZURBAZAR Flowgraph · Models for Multistate Time-to-Event Data IMAN and CONOVER · A Modern Approach to Statistics JACKSON · A User’s Guide to Principle Components JOHN · Statistical Methods in Engineering and Quality Assurance JOHNSON · Multivariate Statistical Simulation JOHNSON and BALAKRISHNAN · Advances in the Theory and Practice of Statistics: A Volume in Honor of Samuel Kotz JOHNSON and BHATTACHARYYA · Statistics: Principles and Methods, Fifth Edition JUDGE, GRIFFITHS, HILL, LU TKEPOHL, and LEE · The Theory and Practice of Econometrics, Second Edition JOHNSON and KOTZ · Distributions in statistics JOHNSON and KOTZ · (editors) Leading Personalities in Statistical Sciences: From the Seventeenth Century to the Present JOHNSON, KOTZ, and BALAKRISHNAN · Continuous Univariate Distributions, Volume 1, Second Edition JOHNSON, KOTZ, and BALAKRISHNAN · Continuous Univariate Distributions, Volume 2, Second Edition JOHNSON, KOTZ, and BALAKRISHNAN · Discrete Multivariate Distributions JOHNSON, KOTZ, and KEMP · Univariate Discrete Distributions, Second Edition JURECKOVA and SEN · Robust Statistical Procedures: Asymptotics and Interrelations JUREK and MASON · Operator-Limit Distributions in Probability Theory KADANE · Bayesian Methods and Ethics in a Clinical Trial Design KADANE and SCHUM · A Probabilistic Analysis of the Sacco and Vanzetti Evidence KALBFLEISCH and PRENTICE · The Statistical Analysis of Failure Time Data, Second Edition KARIYA and KURATA · Generalized Least Squares KASS and VOS · Geometrical Foundations of Asymptotic Inference KAUFMAN and ROUSSEEUW · Finding Groups in Data: An Introduction to Cluster Analysis ∗

∗

Now available in a lower priced paperback edition in the Wiley Classics Library

KEDEM and FOKIANOS · Regression Models for Time Series Analysis KENDALL, BARDEN, CARNE, and LE · Shape and Shape Theory KHURI · Advanced Calculus with Applications in Statistics, Second Edition KHURI, MATHEW, and SINHA · Statistical Tests for Mixed Linear Models ∗ KISH · Statistical Design for Research KLEIBER and KOTZ · Statistical Size Distributions in Economics and Actuarial Sciences KLUGMAN, PANJER, and WILLMOT · Loss Models: From Data to Decisions KLUGMAN, PANJER, and WILLMOT · solutions Manual to Accompany Loss Models: From Data to Decisions KOTZ, BALAKRISHNAN and JOHNSON · Continuous Multivariate Distributions, Volume 1, Second Edition KOTZ and JOHNSON · (editors) Encyclopedia of Statistical Sciences: Volumes 1 to 9 with Index KOTZ and JOHNSON · (editors) Encyclopedia of Statistical Sciences: Supplement Volume KOTZ, READ, and BANKS · (editors) Encyclopedia of Statistical Sciences: Update Volume 1 KOTZ, READ, and BANKS · (editors) Encyclopedia of Statistical Sciences: Update Volume 2 KOVALENKO, KUZNETZOV, and PEGG · Mathematical Theory of Reliability of Time-Dependent Systems with Practical Applications KUROWICKA and COOKE · Uncertainty Analysis with High Dimensional Dependence Modelling LACHIN · Biostatistical Methods: The Assessment of Relative Risks LAD · Operational Subjective Statistical Methods: A Mathematical, Philosophical, and Historical Introduction LAMPERTI · Probability: A Survey of the Mathematical Theory, Second Edition LANGE, RYAN, BILLARD, BRILLINGER, CONQUEST, and GREENHOUSE · Case Studies in Biometry LARSON · introduction to Probability Theory and Statistical Inference, Third Edition LAWLESS · Statistical Models and Methods for Lifetime Data, Second Edition LAWSON · statistical Methods in Spatial Epidemiology, Second Edition LE · Applied Categorical Data Analysis LE · Applied Survival Analysis LEE and WANG · Statistical Methods for Survival Data Analysis, Third Edition LEPAGE and BILLARD · Exploring the Limits of Bootstrap LEYLAND and GOLDSTEIN · (editors) Multilevel Modelling of Health Statistics LIAO · Statistical Group Comparison LINDVALL · Lectures on the Coupling Method LINHART and ZUCCHINI · Model Selection LITTLE and RUBIN · Statistical Analysis with Missing Data, Second Edition LLOYD · The Statistical Analysis of Categorical Data LOWEN and TEICH · Fractal-Based Point Processes MAGNUS and Neudecker · Matrix Differential Calculus with Applications in Statistics and Econometrics, Revised Edition MALLER and ZHOU · Survival Analysis with Long Term Survivors MALLOWS · Design, Data, and Analysis by Some Friends of Cuthbert Daniel MANN, SCHAFER, and SINGPURWALLA · Methods for Statistical Analysis of Reliability and Life Data MANTON, WOODBURY, and TOLLEY · Statistical Applications Using Fuzzy Sets MARCHETTE · Random Graphs for Statistical Pattern Recognition MARDIA and JUPP · Directional Statistics MARKOVICH · Nonparametric Analysis of Univariate Heavy-Tailed Data: Research and Practice MARONNA, MARTIN, and YOHAI · Robust Statistics: Theory and Methods MASON, GUNST, and HESS · Statistical Design and Analysis of Experiments with Applications to Engineering and Science, Second Edition MCCULLOCH and SERLE · Generalized, Linear, and Mixed Models MCFADDEN · Management of Data in Clinical Trials MCLACHLAN · Discriminant Analysis and Statistical Pattern Recognition MCLACHLAN, DO, and AMBROISE · Analyzing Microarray Gene Expression Data MCLACHLAN and KRISHNAN · The EM Algorithm and Extensions ∗

Now available in a lower priced paperback edition in the Wiley Classics Library

MCLACHLAN and PEEL · Finite Mixture Models MCNEIL · Epidemiological Research Methods MEEKER and ESCOBAR · Statistical Methods for Reliability Data MEERSCHAERT and SCHEFFLER · Limit Distributions for Sums of Independent Random Vectors: Heavy Tails in Theory and Practice MICKEY, DUNN, AND CLARK · Applied Statistics: Analysis of variance and Regression, Third Edition ∗ MILLER Survival Analysis, Second Edition MONTGOMERY, PECK, and VINING · Introduction to Linear Regression Analysis, Fourth Edition MORGENTHALER and TUKEY · Configural Polysampling: A Route to Practical Robustness MUIRHEAD · Aspects of Multivariate Statistical Theory MULLER AND STEWART · Linear Model Theory: Univariate, Multivariate, and Mixed Models MURRAY · X-STAT 2.0 Statistical Experimentation, Design Data Analysis, and Nonlinear Optimization MURTHY, XIE, and JIANG Wei · bull Models MYERS · and Montgomery Response Surface Methodology: Process and Product Optimization Using Designed Experiments, Second Edition MYERS, MONTGOMERY, and VINING · Generalized Linear Models, With Applications in Engineering and the Sciences † NELSON · Accelerated Testing, Statistical Models, Test Plans, and Data Analysis † NELSON · Applied Life Data Analysis NEWMAN · Biostatistical Methods in Epidemiology OCHI · Applied Probability and Stochastic Processes in Engineering and Physical Sciences OKABE, BOOTS, SUGIHARA, and CHIU · Spatial Tesselations: Concepts and Applications of Voronoi Diagrams, Second Edition OLIVER and SMITH · influence Diagrams, Belief Nets and Decision Analysis PALTA · Quantitative Methods in Population Health: Extentions of Ordinary Regression PANJER · Operational Risks: Modeling Analytics PANKRATZ · Forecasting with Dynamic Regression Models PANKRATZ · Forecasting with Univariate Box-Jenkins Models: Concepts and Cases ∗ PARZEN · Modern Probability Theory and Its Applications PE NA, TIAO, and TSAY · A Course in Time Series Analysis PIANTADOSI · Clinical Trials: A Methodologic Perspective PORT · Theoretical Probability for Applications POURAHMADI · Foundations of Time Series Analysis and Prediction Theory PRESS · Bayesian Statistics: Principles, Models, and Applications PRESS · Subjective and Objective Bayesian Statistics, Second Edition PRESS and TANUR · The Subjectivity of Scientists and the Bayesian Approach PUKELSHEIM · Optimal Experimental Design PURI, VILAPLANA, and WERTZ · New Perspectives in Theoretical and Applied Statistics PUTERMAN Markov · Decision Processes: Discrete Stochastic Dynamic Programming QIU · Image Processing and Jump Regression Analysis ∗ RAO · Linear Statistical Inference and its Applications, Second Edition RAUSAND and HØYLAND · System Reliability Theory: Models, Statistical Methods and Applications, Second Edition RENCHER · Linear Models in Statistics RENCHER · Methods of Multivariate Analysis, Second Edition RENCHER · Multivariate Statistical Inference with Applications RIPLEY · Spatial Statistics RIPLEY · Stochastic Simulation ROBINSON · Practical Strategies for Experimenting ROHATGI and SALEH · An Introduction to Probability and Statistics, Second Edition ROLSKI, SCHMIDLI, SCHMIDT, and TEUGELS · Stochastic Processes for Insurance and Finance ROSENBERGER and LACHIN · Randomization in clinical Trials: Theory and Practice ROSS · introduction to Probability and Statistics for Engineers and Scientists ∗ †

Now available in a lower priced paperback edition in the Wiley Classics Library Now available in a lower priced paperback edition in the Wiley Interscience Paperback Series

ROSSI, ALLENBY, and MCCULLOCH · Bayesian Statistics and Marketing ROUSSEEUW and LEROY · Robust Regression and Outline Detection RUBIN · Multiple Imputation for Nonresponse in Surveys RUBINSTEIN · Simulation and the Monte Carlo Method RUBINSTEIN and MELAMED · Modern Simulation and Modeling RYAN · Modern Regression Methods RYAN · Statistical Methods for Quality Improvement, Second Edition SALEH · Theory of Preliminary Test and Stein-Type Estimation with Applications SALTELLI, CHAN, and SCOTT · (editors) Sensitivity Analysis ∗ SCHEFFE · The Analysis of Variance SCHIMEK · Smoothing and Regression: Approaches, Computation, and Application SCHOTT Matrix Analysis for Statistics SCHOUTENS · Levy Processes in Finance: Pricing Financial Derivatiyes SCHUSS · Theory and Applications of Stochastic Differential Equations SCOTT · Multivariate Density Estimation: Theory, Practice, and Visualization ∗ SEARLE · Linear Models SEARLE · Linear Models for Unbalanced Data SEARLE · Matrix Algebra Useful for Statistics SEARLE and WILLETT · Matrix Algebra for Applied Economics SEBER · Multivariate Observations SEBER and LEE · Linear Regression Analysis, Second Edition SEBER and WILD · Nonlinear Regression SENNOTT · Stochastic Dynamic Programming and the Control of Queueing Systems ∗ SERFLING · Approximation Theorems of Mathematical Statistics SHAFER and VOVK · Probability and Finance: Its Only a Game! SILVAPULLE and SEN · Constrained Statistical Inference: Inequality, Order, and Shape Restrictions SINGPURWALLA · Reliability and Risk: A Bayesian Perspective SMALL and MCLEISH · Hilbert Space Methods in Probability and Statistical Inference SRIVASTAVA · Methods of Multivariate Statistics STAPLETON · Linear Statistical Models STAUDTE and SHEATHER · Robust Estimation and Testing STOYAN, KENDALL, and MECKE · Stochastic Geometry and Its Applications, Second Edition STOYAN and STOYAN · Fractals, Random and Point Fields: Methods of Geometrical Statistics STYAN · The Collected Papers of T. W. Anderson: 1943–1985 SUTTON, ABRAMS, JONES, SHELDON, and SONG · Methods for Meta-Analysis in Medical Research TANAKA · Time Series Analysis: Nonstationary and Noninvertible Distribution Theory THOMPSON · Empirical Model Building THOMPSON · Sampling, Second Edition THOMPSON · Simulation: A Modeler’s Approach THOMPSON and SEBER · Adaptive Sampling THOMPSON, WILLIAMS, and FINDLAY · Models for Investors in Real World Markets TIAO, BISGAARD, HILL, PE NA, and STIGLER · (editors) Box on Quality and Discovery: with Design, Control, and Robustness TIERNEY · LISP-STAT: An Object-Oriented Environment for Statistical Computing and Dynamic Graphics TSAY · Analysis of Financial Time Series UPTON and FINGLETON · Spatial Data Analysis by Example, Volume II: Categorical and Directional Data VAN BELLE · Statistical Rules of Thumb VAN BELLE, FISHER, HEAGERTY, and LUMLEY · Biostatistics: A Methodology for the Health Sciences, Second Edition VESTRUP · The Theory of Measures and Integration VIDAKOVIC · statistical Modeling by Wavelets VINOD and REAGLE · Preparing for the Worst: Incorporating Downside Risk in Stock Market Investments WALLER and GOTWAY · Applied Spatial Statistics for Public Health Data ∗

Now available in a lower priced paperback edition in the Wiley Classics Library

WEERAHANDI · Generalized Inference in Repeated Measures: Exact Methods in MANOVA and Mixed Models WEISBERG · Applied Linear Regression, Second Edition WELISH · Aspects of Statistical Inference WESTFALL and YOUNG · Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment WHITTAKER · Graphical Models in Applied Multivariate Statistics WINKER · Optimization Heuristics in Economics: Applications of Threshold Accepting WONNACOTT and WONNACOTT · Econometrics, Second Edition WOODING · Planning Pharmaceutical Clinical Trials: Basic Statistical Principles WOOLSON and CLARKE · Statistical Methods for the Analysis of Biomedical Data, Second Edition WU and HAMADA · Experiments: Planning, Analysis, and Parameter Design Optimization WU and ZHANG · Nonparametric Regression Methods for Longitudinal Data Analysis: Mixed-Effects Modeling Approaches YANG · The Construction Theory of Denumerable Markov Processes YOUNG, VALERO-MORA, and FRIENDLY · Visual Statistics: Seeing Data with Dynamic interactive Graphics ∗ ZELLNER · An introduction to Bayesian Inference in Econometrics ZELTERMAN · Discrete Distributions: Applications in the Health Sciences ZHOU, OBUCHOWSKI and McCLISH · Statistical Methods in Diagnostic Medicine

∗

Now available in a lower priced paperback edition in the Wiley Classics Library