Discrete-Time Semi-Markov Random Evolutions and Their Applications [1 ed.] 9783031334283, 9783031334290

Citation preview

Probability and Its Applications

Nikolaos Limnios Anatoliy Swishchuk

Discrete-Time Semi-Markov Random Evolutions and Their Applications

Probability and Its Applications Series Editors Steffen Dereich, Institut für Math Statistik, Universität Münster, Münster, Nordrhein-Westfalen, Germany Davar Khoshnevisan, Department of Mathematics, The University of Utah, Salt Lake City, UT, USA Andreas E. Kyprianou, Department of Statistics, University of Warwick, Coventry, UK Mariana Olvera-Cravioto, Statistics and Operations Research, UNC Chapel Hill, Chapel Hill, NC, USA

Probability and Its Applications is designed for monographs on all aspects of probability theory and stochastic processes, as well as their connections with and applications to other areas such as mathematical statistics and statistical physics.

Nikolaos Limnios • Anatoliy Swishchuk

Discrete-Time Semi-Markov Random Evolutions and Their Applications

Nikolaos Limnios Laboratoire de Mathématiques Appliquées Sorbonne University Alliance Université de Technologie de Compiègne Compiègne, France

Anatoliy Swishchuk Department of Mathematics and Statistics University of Calgary Calgary, AB, Canada

ISSN 2297-0371 ISSN 2297-0398 (electronic) Probability and Its Applications ISBN 978-3-031-33428-3 ISBN 978-3-031-33429-0 (eBook) https://doi.org/10.1007/978-3-031-33429-0 Mathematics Subject Classification: 60F05, 60F17, 60G42, 60H25, 60J05, 60K15, 60K17, 60K37 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This book is published under the imprint Birkhäuser, www.birkhauser-science.com by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

Random evolutions are operator valued stochastic processes in Banach spaces. More precisely, they are abstract mathematical models of stochastic evolutionary systems in random media described by integral equations in Banach spaces. They appeared in the literature for the first time, about 55 years ago, in two papers by Griego and Hersh, [58, 59]. Hersh in his papers, [64, 65], gives an account of the birth and evolution of this theory. Some additional basic works are [30, 45, 66, 67, 124, 146]; see also, [4, 26, 31, 34, 52, 74, 78, 89–91, 104, 105, 123, 125, 145, 156]. For its asymptotic theory within semi-Markov random media, see [90, 91, 157, 166] and references therein. Our aim in this book is to extend the theory and applications of discrete-time random evolutions in semi-Markov random media, i.e., essentially with semiMarkov chains as switching or driving processes. We consider random evolutions in discrete-time underlying by semi-Markov chains in general state space. Discrete-time semi-Markov chains (SMC) are only recently used in applications, especially in DNA-analysis, image and speech processing, finance, insurance, reliability, etc., see [12] and references therein. These applications have stimulated a research effort in this domain. Despite the increasing demand, there are only a few works in the literature on SMC where a large part concerns estimation problems of hidden semi-Markov models. For example, in the Markov processes we have a huge literature in both cases, the discrete-time (Markov chain) and the continuous-time (Markov processes). In the semi-Markov processes, we have a large literature only for the continuous-time (semi-Markov processes or, equivalently, Markov renewal processes). Even if the discrete-time case were introduced in the literature almost in the same period, in 1960s, as for the continuous-time case, in the mid-1950s. After definition of discrete-time semi-Markov random evolution, we propose an asymptotic theory in a functional setting (series scheme). This allows one to obtain limit theorems, for discrete-time stochastic systems, in semi-Markov random media or random environment. Discrete-time random evolution was introduced, for the Markov chains, by Cohen [30] and Keepler [82]. The discrete-time semi-Markov random evolutions have been introduced and studied their asymptotic theory in v

vi

Preface

a series scheme in [101, 104]. Koroliuk and Swishchuk [91], Swishchuk and Wu [166], Anisimov [4], and Koroliuk and Limnios [90] studied discrete-time random evolutions induced by the embedded Markov chains of continuous time semiMarkov processes. This is equivalent to discrete-time Markov random evolution stopped at random time. In this book we present and extend asymptotic results in some additional directions such as reduced random media, controlled processes, optimal stopping, etc. The limit theorems for random evolutions obtained here are abstract limit theorems, and the limit processes are abstract diffusions. We can see their concrete diffusion forms when we apply these results in concrete stochastic systems. So, we could call this limit as pre-diffusions or random evolution diffusion. The main type of asymptotic results given here concerns weak convergence results in the Skorokhod space: • Average approximation or averaging or stochastic Bogoliubov approximation • Diffusion approximation or weak invariance principle • Normal deviations or diffusion approximation with equilibrium In the semi-Markov case, we don’t have semigroup theory as it is in the Markov case. The main tool is the Markov renewal equation, with the important function (measure) called Markov renewal function, together with the Markov renewal theorem. In the discrete-time this function is represented by a finite series of convolution products of the semi-Markov kernel, instead of an infinite series in the continuous-time case. This is an important advantage from an algorithmic point of view. The method of proof of the asymptotic results presented here is the usual one for functional type results, that is: • Convergence of finite-dimensional distribution of the probability measures • Tightness of the law of the random evolutions The main particular stochastic systems concerned by the asymptotic results of random evolutions in the present book are: • Additive functional • Geometric Markov renewal chains • Dynamic systems of the difference equation type all switched by SMC. Applications of the above results concern, epidemics, financial mathematics, stochastic control problems, as well as estimation problems of the stationary distribution of the SMC and U-statistics, see, e.g., [88, 102]. This book can serve applied mathematicians, researchers, master and PhD students, and also advanced students and researchers in Sciences, Engineering, Epidemiology, Financial mathematics and Economy, Control, etc., who are concerned about stochastic modelling of systems. The book is organized as follows.

Preface

vii

Chapter 1 presents basic results, needed in the sequel, on the stochastic processes on Banach spaces: Markov and semi-Markov processes, martingale problem in Banach space, weak convergence in Banach space, reducible invertible operator, etc. Chapter 2 presents the basic theory of general state space semi-Markov chains, as well as the split and merging of their phase space. Chapter 3 presents the discrete-time semi-Markov random evolution, definition, different time scaling of operators for different scheme of approximations, and tightness of probability measures. Chapter 4 presents the weak convergence of random evolution in different schemes: averaging, diffusion approximation, normal deviations, and application of these results for additive functionals of SMC, geometric Markov renewal processes, dynamical systems or difference equations, etc. Chapter 5 presents Discrete-time Semi-Markov Random Evolution (DTSMRE) in reduced media. The semi-Markov chain is considered in an asymptotic split and merging scheme where it is much simpler to study. Chapter 6 presents controlled DTSMRE in asymptotic approximation scheme. The Merton problem and its solution are also presented. Chapter 7 presents an application of DTSMRE, in particular the dynamical systems, in an epidemiology problem. The ordinary differential equations here are considered in discrete-time as difference equations. Chapter 8 presents an optimal stopping problem for Geometric Markov Renewal Process (GMRP) and pricing in financial mathematics for American and European options. An appendix is devoted to the description of Markov chains with general state space. The authors would like to thank Dr Veronika Rosteck from Springer for her excellent collaboration and help. Compiègne, France Calgary, Canada January 2023

Nikolaos Limnios Anatoliy Swishchuk

Contents

1

2

Discrete-Time Stochastic Calculus in Banach Space . . . . . . . . . . . . . . . . . . . . . 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Random Elements and Discrete-Time Martingales in a Banach Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Martingale Characterization of Markov and Semi-Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Martingale Characterization of Markov Chains. . . . . . . . . . . . . . 1.3.2 Martingale Characterization of Markov Processes . . . . . . . . . . 1.3.3 Martingale Characterization of Semi-Markov Processes . . . . 1.4 Operator Semigroups and Their Generators . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Martingale Problem in a Banach Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Weak Convergence in a Banach Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Reducible-Invertible Operators and Their Perturbations. . . . . . . . . . . . . 1.7.1 Reducible-Invertible Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.2 Perturbation of Reducible-Invertible Operators . . . . . . . . . . . . . .

1 1

4 4 5 6 7 11 12 13 13 15

Discrete-Time Semi-Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Semi-Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Classification of States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Markov Renewal Equation and Theorem . . . . . . . . . . . . . . . . . . . . 2.3 Discrete- and Continuous-Time Connection . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Compensating Operator and Martingales. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Stationary Phase Merging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Semi-Markov Chains in Merging State Space . . . . . . . . . . . . . . . . . . . . . . . 2.6.1 The Ergodic Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6.2 The Non-ergodic Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19 19 20 20 26 27 31 31 34 36 36 39 41

1

ix

x

3

Contents

Discrete-Time Semi-Markov Random Evolutions . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Discrete-time Random Evolution with Underlying Markov Chain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Definition and Properties of DTSMRE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Discrete-Time Stochastic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Additive Functionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Geometric Markov Renewal Chains . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Dynamical Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Discrete-Time Stochastic Systems in Series Scheme . . . . . . . . . . . . . . . . 3.6 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

43 43

4

Weak Convergence of DTSMRE in Series Scheme . . . . . . . . . . . . . . . . . . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Weak Convergence Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Diffusion Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 Normal Deviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.4 Rates of Convergence in the Limit Theorems . . . . . . . . . . . . . . . . 4.3 Proof of Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Proof of Theorem 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Proof of Theorem 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Proof of Theorem 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.4 Proof of Proposition 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Applications of the Limit Theorems to Stochastic Systems . . . . . . . . . 4.4.1 Additive Functionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Geometric Markov Renewal Processes. . . . . . . . . . . . . . . . . . . . . . . 4.4.3 Dynamical Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.4 Estimation of the Stationary Distribution . . . . . . . . . . . . . . . . . . . . 4.4.5 U-Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.6 Rates of Convergence for Stochastic Systems . . . . . . . . . . . . . . . 4.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

55 55 56 56 57 58 59 60 60 66 68 68 71 71 73 75 76 77 79 81

5

DTSMRE in Reduced Random Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Definition and Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Average and Diffusion Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Diffusion Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 Normal Deviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Proof of Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Proof of Theorem 5.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Proof of Theorem 5.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Application to Stochastic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.1 Additive Functionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.2 Dynamical Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83 83 84 86 86 88 89 89 90 91 91 92 94

44 46 50 50 52 52 53 54

Contents

5.5.3 Geometric Markov Renewal Chains . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.4 U-Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

95 97 99

Controlled Discrete-Time Semi-Markov Random Evolutions . . . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Controlled Discrete-Time Semi-Markov Random Evolutions. . . . . . . 6.2.1 Definition of CDTSMREs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.3 Dynamic Programming for Controlled Models . . . . . . . . . . . . . . 6.3 Limit Theorems for Controlled Semi-Markov Random Evolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.1 Averaging of CDTSMREs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.2 Diffusion Approximation of DTSMREs . . . . . . . . . . . . . . . . . . . . . 6.3.3 Normal Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Applications to Stochastic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 Controlled Additive Functionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Controlled Geometric Markov Renewal Processes . . . . . . . . . . 6.4.3 Controlled Dynamical Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.4 The Dynamic Programming Equations for Limiting Models in Diffusion Approximation . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Solution of Merton Problem for the Limiting CGMRP in DA . . . . . . 6.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5.2 Utility Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5.3 Value Function or Performance Criterion . . . . . . . . . . . . . . . . . . . . 6.5.4 Solution of Merton Problem: Examples . . . . . . . . . . . . . . . . . . . . . . 6.5.5 Solution of Merton Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Rates of Convergence in Averaging and Diffusion Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Proofs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.1 Proof of Theorem 6.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.2 Proof of Theorem 6.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.3 Proof of Theorem 6.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.4 Proof of Proposition 6.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.8 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

101 101 103 103 104 105

Epidemic Models in Random Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 From the Deterministic to Stochastic SARS Model . . . . . . . . . . . . . . . . . 7.3 Averaging of Stochastic SARS Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 SARS Model in Merging Semi-Markov Random Media . . . . . . . . . . . . 7.5 Diffusion Approximation of Stochastic SARS Models in Semi-Markov Random Media. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

139 139 140 143 145

5.6 6

7

xi

107 108 109 110 110 111 112 114 115 118 119 119 120 121 123 124 126 126 131 134 134 137

147 156

xii

8

Contents

Optimal Stopping of Geometric Markov Renewal Chains and Pricing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 GMRC and Embedded Markov-Modulated (B, S)-Security Markets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 Definition of the GMRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 Statement of the Problem: Optimal Stopping Rule . . . . . . . . . . 8.3 GMRP as Jump Discrete-Time Semi-Markov Random Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Martingale Properties of GMRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Optimal Stopping Rules for GMRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Martingale Properties of Discount Price and Discount Capital. . . . . . 8.7 American Option Pricing Formulae for embedded Markov-modulated (B, S)-Security markets . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 European Option Pricing Formula for Embedded Markov-Modulated (B, S)-Security Markets . . . . . . . . . . . . . . . . . . . . . . . . 8.9 Proof of Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.10 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1 Transition Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Irreducible Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.3 Recurrent Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.4 Invariant Measures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.5 Uniformly Ergodic Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

157 157 159 159 160 161 162 163 166 168 172 173 175 177 177 181 182 184 186

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

Acronyms

DTSMRE RE SMC MRC EMC GMRC CDTSMRE CGMRP CAF CDS HJB DPE DPP DA SDE ARAC RRAC GBM ODE PDE HARA CRRA MPOP

Discrete-Time Semi-Markov Random Evolution Random Evolution Semi-Markov Chain Markov Renewal Chain Embedded Markov Chain Geometric Markov Renewal Chain Controlled Discrete-Time Semi-Markov Random Evolution Controlled Geometric Markov Renewal Processes Controlled Additive Functionals Controlled Dynamical Systems Hamilton–Jacobi–Bellman (equation) Dynamic Programming Equation Dynamic Programming Principle Diffusion Approximation Stochastic Differential Equation Absolute Risk Aversion Coefficient Relative Risk Aversion Coefficient Geometric Brownian Motion Ordinary Differential Equation Partial Differential Equation Hyperbolic Absolute Risk Aversion Constant Relative Risk Aversion Merton Portfolio Optimization Problem

xiii

Notation

N .Z .Q .R .R+ .fx  .v .δa .(, F , P) .1A or .1(A) .EX .V arX .k .F = (Fk , k ≥ 0) . .(E, E ) .P (x, B) P .q(x, B, k) .R0 .π .ρ .



.

Ej m(x) m .zk , k ≥ 0 .xn , n ≥ 0 .τn , n ≥ 0 . .

the set of natural numbers, .{0, 1, 2, . . .}, and .N∗ = {1, 2, . . .} the set of relative integers the set of rational numbers the set of real numbers the set of real positive numbers .[0, +∞) distribution function of the sojourn time in state .x ∈ E the transpose of vector .v ∈ Rd Dirac measure in .a ∈ Rd probability space indicator function of set A mathematical expectation of X variance of X random evolution, .k ∈ N filtration stochastic basis . = (, F , F, P) state space, a measurable space transition kernel of Markov chain transition operator associated to .P (x, B) semi-Markov kernel, .x ∈ E, B ∈ E , k ∈ N potential operator of the generator Q stationary probability of (semi-) Markov chain stationary probability of the (embedded) Markov chain .xn , n ≥ 0 projection operator on the null space of the operator Q, .Q = P −I a class of split state space .E = ∪dj =1 Ej mean sojourn time in state x mean times, .ρ-mean: .m = E ρ(dx)m(x) semi-Markov chain embedded Markov chain of a semi-Markov chain .zk , k ∈ N jump times of a semi-Markov chain, .τ0 = 0 xv

xvi

θn , n ≥ 1 νk , k ≥ 0 .τ (k) . M

.a ∨ b .a ∧ b .[x] .xn .ϑ . .

o(x) O(x) .ϕ . .

a.e. a.s. r.v.(s) i.i.d. a.s. .−→ P

−→

.

d

Notation

inter-jump times of a semi-Markov chain counting process of jumps of the semi-Markov chain (.zk ) the time of the last jump before k, .τ (k) = τνk square characteristic of the martingale M maximum of a and b minimum of a and b the integer part of the real number x increment of the sequence .(xn ), .xn = xn − xn−1 the shift operator on the sequences: .ϑ(x0 , x1 , . . . ) = (x1 , x2 , . . . ) small o of x in .x0 ∈ R, .limx→x0 o(x)/x = 0 big O of x in .x0 ∈ R: .limx→x0 O(x)/x = c ∈ R a test function; .ϕ  (u), ϕ  (u), . . . are derivatives of function .ϕ,   .ϕu (u, v), .ϕuu (u, v), . . . partial derivatives with respect to the first variable almost everywhere almost surely random variable(s) independent and identically distributed r.v.s almost sure convergence convergence in probability

−→

convergence in distribution / law

−→ .⇒

convergence in norm .Lp weak convergence in the sense of Skorokhod topology in the space .D[0, ∞) or in Sup norm in .C[0, ∞). the Banach space of real measurable bounded functions defined on E the space of real-valued continuous bounded functions defined on E the space of continuous functions on E vanishing at infinity the space of continuous functions on E having continuous derivatives of order up to and including k vanishing at infinity the space of bounded continuous functions on E having continuous derivatives of all orders vanishing at infinity the space of k-th times continuously differentiable functions on E with compact support the real-valued function differentiable with respect to the first variable and continuous to the second variable the space of twice differentiable function in the first argument and continuous in the second the space of E-valued continuous functions defined on .R+ the space of E-valued right continuous functions having left limits (cadlag), defined on .R+ the domain of the operator Q the positive and negative part of function g respectively

.

Lp

.

B C(E)

.

C0 (E) C0k (E)

. .

C0∞ (E)

.

Cκk (Rd )

.

C 1,0 (E × F )

.

C 2,0 (Rd × E)

.

CE [0, ∞) DE [0, ∞)

. .

DQ , .D(Q) g+ , .g−

. .

Chapter 1

Discrete-Time Stochastic Calculus in Banach Space

1.1 Introduction In this chapter, we introduce some preliminary definitions, notions, and results which are necessary for our later analysis of discrete-time semi-Markov random evolutions in series scheme. We give a description of random elements and discretetime martingales in Banach space (Sect. 1.2), martingale characterization of Markov and semi-Markov processes and chains (Sect. 1.3), operator semi-groups and their generators (Sect. 1.4), martingale problem in a Banach space (Sect. 1.5), weak convergence in a Banach space (Sect. 1.6), and perturbations of reducible-invertible operators (Sect. 1.7) (see [2, 40, 156–158]). While the main object of this book is the random evolution in discrete-time, we present also, here, continuous-time processes since they appear as weak limit of the previous ones. For Markov chains, the reader can see in Appendix A.

1.2 Random Elements and Discrete-Time Martingales in a Banach Space We consider .(Ω, F , (Fn , n ≥ 0), P) to be a stochastic basis (a filtered space) and (B, B, ·) to be a separable Banach space, over the field of real numbers .R, with ∗ .σ -algebra of Borel sets .B and with norm .· . Let .B be a dual space to .B separating the points of .B (that is, for any .x1 , x2 ∈ B, such that .x1 = x2 , there is a functional ∗ . ∈ B , such that .(x1 ) = (x2 )). We use the following notations: .FB [0, T ] for a space of functions on .[0, T ] with values in .B, .CB [0, T ] for a space of continuous functions on .[0, T ] with values in .B, and .DB [0, T ] for a space of right-continuous and having left limits functions (càdlàg) on .[0, T ] with values in .B. Here, .0 < T < ∞. .

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0_1

1

2

1 Discrete-Time Stochastic Calculus in Banach Space

Let .V := V (ω) map the probability space .Ω into the Banach space .B; .V : Ω → B. If V is measurable (that is, .V −1 B ⊂ F ), then V is called a random element. If .B = R1 := R, then V is called a random variable; if .B = Rd , .d > 1, then V is called a random vector; and if .B = C[0, T ] or .B = D[0, T ], then V is called a random process or a stochastic process or a random function. A random element V is called a weak random element if the map V is a weakly measurable (as the map .(V ) : Ω → R,  ∈ B∗ ) element from .Ω to .B. A random element V is called a strong random element if the map V is a strongly measurable element from .Ω to .B, that is, V is .F /B-measurable. Distribution of random element V is a probability measure .μ := P ◦ V −1 on .(B, B). It means that .(B, B, μ) is a probability space. The distribution .μ is always taken on a metric space in contrast to the probability measure .P defined on a space of any nature. For any .ω ∈ Ω, we denote by .Vn (ω), n ∈ N, the element of .FB [0, T ]. The random sequence .Vn , n ∈ N, is said to be adapted to a filtration .Fn , if for each n the random element .Vn is .Fn -measurable, that is, {Vn (ω) ∈ A} ∈ Fn ,

.

n ∈ N,

A ∈ B.

A random element V is integrable in the Bochner sense if there exists a sequence Vn (ω) of simple functions in .B which converges to .V (ω) a.s., and

.

 .

lim

m,n→∞ Ω

Vm (ω) − Vn (ω) dP = 0.

It is worth noticing here that .L1 (Ω, F , Fn , P; B) = E , for the norm .·, where .E is the set of simple function on B.  Then the .limn→∞ Ω Vn (ω)dP exists, and by definition 

 V (ω)dP := lim

.

n→∞ Ω

Ω

Vn (ω)dP.

This limit defines the strong expectation .EV of .V :  EV :=

V (ω)dP(ω).

.

Ω

 Compare with the weak expectation: .(EV ) = E(V ) := Ω (V (ω))dP(ω), . ∈ B∗ . We note that if .Vt (ω) ∈B for any .t ∈ R+ := [0, ∞), then we can similarly define the Bochner integral . R+ Vs (ω)ds in .B.

1.2 Random Elements and Discrete-Time Martingales in a Banach Space

3

Let V be integrable in the Bochner sense and .G -measurable, .σ -algebra .G ⊂ F . Then there exists a function .E[V | G ] : Ω → B which is integrable in the Bochner sense, strongly measurable with respect to .G , unique a.s., and 

 V (ω)dP(ω) =

.

A

E[V | G ]dP(ω),

A ∈ G.

A

Function .E[V | G ] is called a strongly conditional expectation of V with respect to .G . The main properties of .E[V | G ] are similar to those for regular real-valued conditional expectations. The weak conditional expectation is defined as .(E[V | G ]), for any . ∈ B∗ . By analogy, we can define the strong conditional expectation for random sequence .Vn (ω) with respect to filtration .Fn . Let .Mn be a map from .Ω to .B, which is integrable in the Bochner sense and strongly measurable with respect to .Fn . A sequence .Mn is called a strongly .Fn martingale in .B, if 1. .E Mn  < ∞. 2. .E[Mn | Fm ] = Mm , .m < n. A sequence .Mn (ω) is called a weak .Fn -martingale in .B, if 1. above is satisfied and 2’. .E[(Mn ) | Fm ] = (Mm ), .m < n, . ∈ B∗ . Namely, .Mn is a weak .Fn -martingale in .B, if .(Mn ) is a real .Fn -martingale with respect to .Fn , for any . ∈ B∗ . We call .Mn then a discrete-time martingale. We note that strong (weak) semimartingales (supermartingales and submartingales) are defined quite similarly by analogy with real-valued cases. Let .τ be a stopping time with respect to .Fn , that is, .{τ < n} ∈ Fn . The process .Mn is said to be a local martingale if there exists a sequence of stopping times .τk , k ∈ N, such that .τk → ∞, and for any k, .Mn∧τk is an .Fn -martingale, where .n ∧ τk := min(n, τk ). A martingale .Mn in .B is called a square integrable martingale if .

sup E Mn 2 < ∞. n∈N

Quadratic variation of a martingale .Mn in .B is defined by a quadratic variation of real-valued martingale .(Mn ), for any . ∈ B∗ . It is a weak quadratic variation. We use the following symbol for quadratic variation: . (Mn ) . In this way, .

(Mn ) :=

n 

E[2 (Mk − Mk−1 ) | Fk−1 ],

k=1

where .Fk := σ ((Ms ); 0 ≤ s ≤ k), .k ∈ N. We often use another definition of quadratic variation. If the process .2 (Mn ) − vn is an .Fn -martingale in .R, then the real-valued random sequence .vn is a quadratic

4

1 Discrete-Time Stochastic Calculus in Banach Space

variation of .(Mn ) : (Mn ) = vn . If there exists the process .Vn in .B such that (Vn ) = vn , for any . ∈ B∗ , then .Vn is a Banach-valued analogue of the quadratic variation for the martingale .Mn in .B. By definition, the quadratic variation . (Mn ) satisfies the relation

.

E[2 (Mn − Mm ) | Fk ] = E[ (Mn ) − (Mm ) | Fk ].

.

Note the important inequality for martingales: E

.

sup

n∈{1,2,...,N }

 |(Mn )| ≤ 3E (Mn ) ,

 ∈ B∗ .

From this, we have that if . (Mn ) = 0 for any . ∈ B∗ , then .(Mn ) = 0 and .Mn = 0 a.s. because the space .B∗ separates points of .B.

1.3 Martingale Characterization of Markov and Semi-Markov Chains We now describe martingale properties of Markov chains, Markov processes, and semi-Markov processes and consider merging of semi-Markov processes (see [41, 91]).

1.3.1 Martingale Characterization of Markov Chains Let .Y := (yn ; n ∈ N) be a homogeneous Markov chain on a measurable phase space .(E, E ) with stochastic kernel .P (y, A), y ∈ E, A ∈ E . Let P be the operator on the space .B(E)  P ϕ(y) :=

P (y, dz)ϕ(z) = E[ϕ(yn )|yn−1 = y] := Ey [ϕ(y1 )],

.

(1.1)

E

generated by .P (y, A), and .FnY := σ {yk ; 0 ≤ k ≤ n} be the natural filtration of the process .(yn , n ∈ N). The Markov property can be described by Y E[ϕ(yn )|Fn−1 ] = P ϕ(yn−1 ).

.

(1.2)

This is a consequence of (1.1) and of the Markov property of the chain .(yn , n ∈ N) Y E[ϕ(yn ) | Fn−1 ] = E[ϕ(yn ) | yn−1 } = P ϕ(yn−1 ).

.

(1.3)

1.3 Martingale Characterization of Markov and Semi-Markov Chains

5

We note that P ϕ(yn ) − ϕ(y) =

n−1 

.

[P − I ]ϕ(yn ),

y0 = y.

(1.4)

k=0

Hence, from (1.3) and (1.4), it follows that E(ϕ(yn ) − ϕ(y) −

.

n−1  [P − I ]ϕ(yn )|yn−1 = y) = 0, k=0

or E(ϕ(yn ) − ϕ(y) −

.

n−1  Y [P − I ]ϕ(yn )|Fn−1 ) = 0. k=0

Consequently, Mn := ϕ(yn ) − ϕ(y) −

.

n−1  [P − I ]ϕ(yk )

(1.5)

k=0

is an .FnY -martingale. The quadratic variation

.

M n :=

n−1 

Y E[(Mk − Mk−1 )2 |Fk−1 ]

k=1

of the martingale .Mn in (1.5) is given by

.

M n =

n−1  [P ϕ 2 (yk ) − (P ϕ(yk ))2 ].

(1.6)

k=0

1.3.2 Martingale Characterization of Markov Processes Let .Y := (y(t); t ∈ R+ ) be a homogeneous Markov process on a measurable phase space .(E, E ) with transition probabilities .P (t, y, A), .t ∈ R+ , .y ∈ E, .A ∈ E . The transition probabilities .P (t, y, A) generate the contraction semigroup .Γ (t) on the Banach space .B(E) by the following formula:  Γ (t)f (y) :=

P (t, y, dz)ϕ(z) = E[ϕ(y(t)) | y(0) = y].

.

E

(1.7)

6

1 Discrete-Time Stochastic Calculus in Banach Space

Let Q be the infinitesimal operator of the Markov process .(y(t))t∈R+ . Then  Γ (t)ϕ(y) − ϕ(y) =

.

t



t

QΓ (s)ϕ(y(s))ds =

(1.8)

Γ (s)Qϕ(y(s))ds.

0

0

From this and the Markov property, it follows that 

t

E[ϕ(y(t)) − ϕ(y) −

.

0

Qϕ(y(s))ds|FsY ] = 0,

(1.9)

where FsY := σ {(y(u)); 0 ≤ u ≤ s}.

.

Therefore, 

t

M(t) := ϕ(y(t)) − ϕ(y) −

Qϕ(y(s))ds

.

(1.10)

0

is .FtY -martingale. The quadratic variation of .m(t) is  .

M (t) :=

t

[Qϕ 2 (y(s)) − 2ϕ(y(s))Qϕ(y(s))]ds.

(1.11)

0

1.3.3 Martingale Characterization of Semi-Markov Processes Let .y(t) := yν(t) be a continuous-time semi-Markov process, constructed by Markov renewal process .(yn , θn )n∈N on .E ×R+ and .γ (t) := t −τν(t) be a backward recurrence time process. Then the process .(y(t), γ (t), t ∈ R+ ) on .E × R+ is a homogeneous Markov process with the generator fy (t) d  Qϕ(y, t) = ϕ(y, t) + [P ϕ(y, 0) − ϕ(y, t)], dt F y (t)

.

ϕ ∈ C 0,1 (E × R+ ). (1.12)

(t, (y, s), ·) of the Let .Γ(t) be the semigroup generated by transition probabilities .P process .(y(t), γ (t), t ∈ R+ ). Then Γ(t)ϕ(y, 0) − ϕ(y, 0) =



t

.

0

Γ(s)ϕ(y(s), γ (s))ds. Q

(1.13)

1.4 Operator Semigroups and Their Generators

7

This, together with the Markov property, implies 

t

E[ϕ(y(t), γ (t)) − ϕ(y, 0) −

.

s ] = 0,  Qϕ(y(s), γ (s))ds | F

(1.14)

0

where s := σ {y(u), γ (u); 0 ≤ u ≤ s}. F

.

Consequently,  := ϕ(y(t), γ (t)) − ϕ(y, 0) − M(t)



t

.

 Qϕ(y(u), γ (u))du

(1.15)

0

t - martingale. is an .F    (t) of the martingale .M(t)  in (1.15) is given by The quadratic variation . M

.

   (t) = M



t

 2 (y(u), γ (u)) − 2ϕ(y(u), γ (u))Qϕ(y(u),  [Qϕ γ (u))]du.

(1.16)

0

1.4 Operator Semigroups and Their Generators Let .(B, B, ·) be a real separable Banach space .B with .σ -algebra of Borel sets .B and the norm .· . A one-parameter family .(Γ (t); t ∈ R+ ) of bounded linear operators on .B is called a semigroup of operators (see [41, 45, 69]), if (i) .Γ (0) = I is the identity operator and (ii) Γ (t + s) = Γ (t)Γ (s),

.

(1.17)

for all .s, t ≥ 0. The semigroup .(Γ (t), t ∈ R+ ) is said to be a contraction semigroup if .Γ (t) ≤ 1, for all .t ≥ 0. It will be said a strongly continuous semigroup or equivalently a .C0 -semigroup, if (iii) .

lim (Γ (t) − I )f  = 0,

t→0

f ∈ B.

8

1 Discrete-Time Stochastic Calculus in Banach Space

The generator (or infinitesimal operator) of a semigroup .(Γ (t), t ∈ R+ ) is the linear operator A, defined by Af := lim t −1 [(Γ (t) − I )f ],

(1.18)

.

t→0

where the limit is considered in the strong sense and its domain .D(A) is defined by D(A) := {f ∈ B : lim t −1 [(Γ (t) − I )f ]

.

t→0

exists}.

The following result is well known (see, e.g., [18, 40]). Theorem 1.1 Let us consider a strongly continuous semigroup .(Γ (t), t ≥ 0) on a Banach space .B and A its generator with domain .D(A). Then, (i) .D(A) is a dense subspace of .B, that is, .D(A) = B. (ii) A is a closed linear operator in .B. n (iii) . ∞ n=1 D(A ) = B. Also, for a strongly continuous semigroup .(Γ (t), t ∈ R+ ) on .B with the generator .A, we have

dΓ (t) .

dt = Γ (t)A = AΓ (t), Γ (0) = I.

(1.39)

We now give a few examples of semigroup operators and their generators. Example 1.1 (Exponential semigroup.) Let A be a bounded linear operator on .B. Define the following one-parameter family: Γ (t) = etA :=

∞ k  t

.

k=0

k!

Ak ,

t ≥ 0.

It is easy to verify that .(Γ (t), t ∈ R+ ) defined above is a strongly continuous semigroup with generator .A. Here .D(A) = B. Example 1.2 (Uniform motion on the real line.) Let .B = C(R) be a Banach space of bounded continuous functions on .R equipped with .sup-norm, and let .Γ (t)ϕ(x) := ϕ(x + υt), where .υ > 0 is a constant velocity, .ϕ ∈ C(R).  1 .(Γ (t), t ∈ R+ ) is a semigroup with .Aϕ(x) = v · ϕ (x) and .D(A) = C (R) is the space of differentiable functions on R with continuous derivatives. Example 1.3 (Motion with velocity depending on the state.) Let .q(t, z) solve the Cauchy problem: .

dq(t, z) = υ(q(t, z)), dt

q(0, z) = z.

1.4 Operator Semigroups and Their Generators

9

Then Γ (t)ϕ(z) := ϕ(q(t, z)),

ϕ ∈ C(R),

.

gives a strongly continuous contraction semigroup and its generator is Aϕ(z) = υ(z)ϕ  (z),

.

ϕ ∈ C 1 (R).

The semigroup property follows from the equality q(t + s, z) = q(s, q(t, z)),

.

z ∈ E,

s, t ∈ R+ .

Example 1.4 (Continuous-time Markov process and its infinitesimal matrix) Here, Γ (t) = P(t) is the infinite dimensional matrix with

.

P(t) = (pij (t); i, j = 1, 2, . . .),

t ≥ 0,

.

and A = Q = (qij ; i, j = 1, 2, . . .).,

.

(1.19)

where Q is the intensity matrix of the Markov process. Example 1.5 (Bienaymé–Galton–Watson Branching process) For a BGW process, we have 

yn−1

yn =

.

ξj ,

n ≥ 1,

and

y0 = 1,

j =1

where .(ξj , j ≥ 1) is an i.i.d. sequence of r.v., we have a semigroup P defined by x  P ϕ(x) = Eϕ( ξj )

.

j =1

and the discrete generator .L = P − I . Example 1.6 (Diffusion processes) Let .y(t) be a diffusion process with drift coefficient .a(t, y) ≡ a(y) and diffusion coefficient .σ (t, y) ≡ σ (y). As these are independent of t, we obtain the so-called homogeneous diffusion process with transition probabilities .P (t, y, A), .t ∈ R+ , y ∈ R, A ∈ B. The associated semigroup and its generator are  Γ (t)ϕ(z) :=

ϕ(y)P (t, z, dy),

.

Y

ϕ ∈ C(R),

(1.20)

10

1 Discrete-Time Stochastic Calculus in Banach Space

and Aϕ(z) = a(y)

.

dϕ(z) 1 2 d 2 ϕ(z) + σ (y) , dz 2 dz2

ϕ ∈ C 2 (R).

(1.21)

Example 1.7 (Wiener process) A standard Brownian motion or, equivalently, a standard Wiener process .w(t), t ≥ 0, that is .w(t) = 0 (a.s.), it has independent increments, and for any .0 ≤ s < t, the r.v. .w(t)−w(s) has the Gaussian distribution with mean 0 and variance .t − s. This is a special case of diffusion. In this special case of a standard Wiener process, we have the transition probability P (t, y, A) = √



1

e−

.

2π t

(z−y)2 2t

dz

(1.22)

A

and Aϕ(z) =

.

1 d 2 ϕ(z) . 2 dz2

It is worth noticing that the transition probability (1.22) of the standard Brownian motion generates a strongly continuous semigroup .Γ (t) on the Banach space .B = C0 (R) of continuous (bounded) functions on .R which vanish at infinity. This is clear from the following relation: 1 .(Γ (t) − I )ϕ(x) = √ 2π

 R

e−y

2 /2

√ [ϕ(y t + x) − ϕ(x)]dy.

In the case where .B is the space of the real-valued measurable bounded functions on .R, then, the transition probability (1.22) does not generate a strongly continuous semigroup. Example 1.8 (Jump Markov process) For a regular homogeneous jump Markov process, the semigroup defined by  Γ (t)ϕ(y) =

P (t, y, dz)ϕ(z),

.

(1.23)

E

where .ϕ ∈ C 1 (R), is a strongly continuous contraction semigroup, with the generator  Aϕ(y) = λ(y)

P (y, dz)[ϕ(z) − ϕ(y)],

.

ϕ ∈ C(R) =: D(A),

E

where .λ(y), .y ∈ E, is the intensity of jumps function.

(1.24)

1.5 Martingale Problem in a Banach Space

11

Example 1.9 (Semi-Markov process) Let .y(t) := yν(t) be the semi-Markov process introduced in Sect. 1.4. As we mentioned earlier, each of the auxiliary processes + .θ (t), γ (t), and .γ (t) compliments .y(t) to a Markov process. In particular, for .γ (t) := t − τν(t) , .(y(t), γ (t)) is a Markov process on .E × R+ with the generator Aϕ(y, t) =

.

dϕ(y, t) + λy (t)[P ϕ(y, 0) − ϕ(y, t)], dt

(1.25)

where fy (t) :=

.

dFy (t) , dt

F y (t) := 1 − Fy (t),

λy (t) :=

fy (t) F y (t)

,

ϕ ∈ C 0,1 (E × R+ ).

The function .λy (t) is the intensity of jumps from state .y ∈ E to the set .E \ {y} at time t. We should mention that a semi-Markov process .y(t) does not generate a semigroup due to the arbitrary distribution function for the sojourn times, rather than the exponential one in the case of a Markov process.

1.5 Martingale Problem in a Banach Space Let .(B, B, ·) be a real separable Banach space, and let .B∗ be a dual space which separates points of .B; let .CB [0, ∞) be a space of continuous bounded functions on .[0, ∞) with values in .B. The solution of the martingale problem for processes .Mt and .Vt in .B is a probability measure .μv on .CB [0, ∞), v ∈ B (or, equivalently, a continuous process .V ∈ B), which satisfies the following conditions (see [91]): 1. .μv (V0 = v) = 1. 2. .mt := (Mt ) is a continuous martingale in .R+ , . ∈ B∗ . 3. .(mt )2 − vt is a continuous martingale in .R+ , . ∈ B∗ , where .vt := (Vt ). Martingale problem for process .Vt can be formulated using some operator A on D(A) ⊂ B. By solution of the martingale problem for A, we mean a measurable stochastic process .Vt ϕ such that for each .ϕ ∈ D(A),

.

 Vt ϕ −

t

Vs Aϕds

.

0

is a martingale with respect to the filtration .Ft , .t ≥ 0. A probability measure P is a solution of the martingale problem for A if the coordinate process defined on .(DB [0, ∞), DB , P ) by .Vt (ω) := ω(t), ω ∈ DB [0, ∞) is a solution of the martingale problem for A defined above, where .DB is the .σ -algebra of the Borel sets of .DB [0, ∞).

12

1 Discrete-Time Stochastic Calculus in Banach Space

Example 1.10 Let .x(t) be a homogeneous Markov process with generator A on C(E), where E is a state space for .x(t). Define .Vt ϕ(x) = ϕ(x(t)), for any .ϕ ∈ C(E) =: B. Then process .Vt is the solution of the martingale problem for operator A, since

.



t

Vt ϕ(x) −

.

 Vs Aϕ(x)ds = ϕ(x(t)) −

0

t

Aϕ(x(s))ds 0

is an .Ft -martingale, .ϕ ∈ D(A) = C(E).

1.6 Weak Convergence in a Banach Space Let .(B, B, μ, ·) be a separable Banach space with .μ a probability measure. In this way, .μ is a probability measure on .B (see [16] and [91]). Probability measures .μn converge weakly to a probability measure .μ (.μn ⇒ μ) if and only if 

 ϕdμn →

.

B

ϕdμ,

as

n→∞

B

for any bounded real-valued continuous function f on .B, that is, .f ∈ C(B). A probability measure .μ on .(B, B) is called tight, if for any .ε > 0 there exists a compact set .Kε such that μ(Kε ) > 1 − ε.

.

Ulam’s theorem states that any probability measure on a Polish space (i.e., a complete separable metric space) is tight. A sequence .Vn := Vn (ω) of random elements converges in distribution to a random element .V , .Vn ⇒ V , as .n → ∞, if distribution .μn of elements .Vn converges weakly to the distribution .μ of element .V : .μn ⇒ μ. A sequence .Vn := Vn (ω) converges in probability to .a ∈ B, .Vn ⇒ a, if for any .ε > 0 P(Vn − a ≥ ε) → 0,

.

as

n → ∞.

By analogy with the real-valued case, the following result holds: if .Vn ⇒ V and P

Vn − Wn  → 0, then .Wn ⇒ V . A family of probability measures .μn on .(B, B) is called weakly compact (or relatively compact), if any sequence of elements .νn contains a weakly converging subsequence. The relationship between tightness and weak compactness is established by the following results: (1) if .μn is tight, then it is weakly compact for any metric space

.

1.7 Reducible-Invertible Operators and Their Perturbations

13

and (2) if metric space is separable and complete, then weak compactness implies tightness. In the case of our space .B, the notions of tightness and weak compactness coincide. We can state the criteria of weak compactness for processes with values in .B with the help of the following result, which we shall use frequently. Let .B∗ be a dual space to .B which separates points of .B and .B∗0 be a dense set in .B∗ . Then a family of processes .Vε (t), ε > 0, in .B is a weakly compact with the limit points in .CB [0, ∞) if and only if the following conditions hold: 1. For any .Δ > 0 and any .T > 0, there exists a compact set .KTΔ ⊆ B : .

lim inf P(Vε (t) ∈ KTΔ ; 0 ≤ t ≤ T ) ≥ 1 − Δ. ε→0

2. For any . ∈ B∗0 , the family of processes .(Vε (t)), ε > 0, is a weakly compact with the limit points in .CR [0, ∞). The condition 1. is called the compact containment criterion (CCC).

1.7 Reducible-Invertible Operators and Their Perturbations 1.7.1 Reducible-Invertible Operators Introduce some necessary notations: .B∗ is a dual space to the Banach space .B, .Q∗ is the linear operator in .B∗ which is adjoint to .Q : .Q∗ (f ) = (Qf ), . ∈ B∗ , f ∈ B, and we define .D ⊥ := { ∈ B∗ : (f ) = 0; ∀f ∈ D ⊂ B} as the family of functionals orthogonal co-vectors from D (see Koroliuk V. and Turbin A. (1983), [91], and [90]). Let us also introduce the following subspaces: .N(Q) := {f ∈ B : Qf = 0} null space of the operator .Q, .R(Q) := {φ ∈ B : Qf = φ}—the space of values of the operator .Q. A linear bounded operator Q is called reducible-invertible if B = N(Q) ⊕ R(Q),

.

dim N(Q) ≥ 1.

Here, .dim N(Q) is the dimension of the space .N(Q). The last decomposition corresponds to the projector .Π onto the null space .N(Q) parallel to .R(Q):

Πf =

.

f, f ∈ N(Q), 0, f ∈ R(Q).

14

1 Discrete-Time Stochastic Calculus in Banach Space

In this case, .I − Π is the projector onto the subspace of values .R(Q):

[I − Π ]f =

.

0, f ∈ N(Q), f, f ∈ R(Q).

The last decomposition in vector form can also be represented as follows: f = Πf + (I − Π )f.

.

It is well known that the adjoint operator .Q∗ is also reducible-invertible .B∗ = N (Q∗ ) ⊕ R(Q∗ ), and the following relations hold: N(Q∗ ) = R(Q)⊥ ,

.

R(Q∗ ) = N(Q)⊥ .

A linear operator Q is called normally solvable if the equation .Qf = φ is solvable for all .φ ∈ R(Q). We note (see above) that the reducible-invertible operator Q is normally solvable and .(φ) = 0, for all . ∈ N(Q∗ ) such that .Q∗  = 0. The potential .R0 of a reducible-invertible operator Q is defined by the relation R0 = (Q + Π )−1 − Π.

.

The invertibility of the operator .Q + Π follows from the fact that its range of values coincides with the whole .B. In fact, if .f ∈ N(Q), then .(Q + Π )f = f, and if .f ∈ R(Q), then .(Q + Π )f = Qf ∈ R(Q). Let us now state the basic properties (which follow from the definitions) of operators .P , Q, and .R0 , which will be used in what follows: 1. 2. . 3. 4.

QΠ Π R0 QR0 QR0k

= Π Q = 0. = R0 Π = 0. = R0 Π = I − Π. = R0 Q = R0k−1 , k > 1.

The third property implies that the potential .R0 is the inverse operator to the operator Q on the subspace of values of .R(Q). Hence, the potential .R0 can be naturally called the reducible inverse operator to the operator .Q. In this case, the general solution of equation .Qf = φ can be represented in the following way: f = R0 φ + f0 ,

.

f0 ∈ N(Q).

It is worth noticing that under condition .Πf = 0, equation .Qf = φ has the unique solution which can be represented in the form f = R0 φ,

.

Πf = 0.

1.7 Reducible-Invertible Operators and Their Perturbations

15

1.7.2 Perturbation of Reducible-Invertible Operators A variety of problems of the asymptotic analysis of stochastic systems can be reduced to the problem of singular perturbation of reducible-invertible operators (see Koroliuk V. and Turbin A. (1983) and [91], [90]). Let Q be a bounded reducible-invertible operator on the Banach space .B : .B = N(Q) ⊕ R(Q) (see the above section for notations). The problem of asymptotic singular perturbation of a reducible-invertible operator Q with small parameter .ε > 0 and perturbing operator .Q1 is formulated in the following way. We have to construct the vector .φ ε = φ + εφ1 which realizes the asymptotic representation [ε−1 Q + Q1 ]φ ε = ψ + εvε

.

(1.26)

for some given vector .ψ and with uniformly bounded in norm vector .vε : .vε  ≤ C < ∞ as .ε → 0. Such a problem arises as the problem of asymptotic solution of the equation [Q + εQ1 ]φ ε = ψ ε

.

for a given vector .ψ ε . The solution of a singular perturbation problem (1.26) is based on the properties of reducible-invertible operators which have been given in the previous section. The left-hand side of the latter equation can be represented in the following form: [ε−1 Q + Q1 ](φ + εφ1 ) = ε−1 Qφ + [Qφ1 + Q1 φ] + εQ1 φ1 .

.

We set

.

Qφ = 0, Qφ1 + Q1 φ = ψ, Q1 φ1 = vε ,

to resolve the right-hand side of the above equation. The first equation means that φ ∈ N(Q). The third equation means that vector .vε = Q1 φ1 is independent of .ε. The main problem is to solve the equation .Qφ1 = ψ − Q1 φ, which we get from the second equality above with given vectors .ψ and .φ. The solvability condition with the reducible-invertible operator Q for the last equation is .Π (ψ − Q1 φ) = 0, where .Π is the projector to .N(Q). Taking into account that .φ ∈ N(Q), that is, .Π φ = φ, the latter condition leads to the equality .Π Q1 Π φ = Π ψ. We note that operator .Π Q1 Π acts in the subspace .N(Q) and .Π Q1 Πf = 0, if .f ∈ R0 . If the operator .Π Q1 Π is invertible, then the equation .

16

1 Discrete-Time Stochastic Calculus in Banach Space

Π Q1 Π φ = Π ψ has a solution with respect to .φ. If operator .Π Q1 Π is not invertible, then we have two cases: .Π Q1 Π is the zero operator or, in other terms,

.

Π Q1 Π = 0.

.

This equation is called a balance condition. Another case is when the operator Π Q1 Π is reducible-invertible, which was considered already above. We note that the solution of the equation .Qφ1 = ψ − Q1 φ has the form:

.

φ1 = R0 (ψ − Q1 φ),

.

Π φ1 = 0,

where .R0 is the potential of operator .Q. Finally, the vector .vε has the following representation: vε = Q1 φ1 = Q1 R0 (ψ − Q1 φ).

.

We state two main results here for the singular perturbation problem (1.26) with invertible operator Q. Proposition 1.1 (.Π Q1 Π = 0) Let the following conditions be satisfied: (i) Bounded operator Q on the Banach space .B is reducible-invertible with projector .Π to the null space .N(Q) := {φ : Qφ = 0}, dim N(Q) ≥ 1. (ii) The perturbing operator .Q1 on .B is closed with a dense domain .B0 ⊆ B, B0 = B. (iii) The contraction operator .Π Q1 Π has the inverse operator .(Π Q1 Π )−1 . Then the asymptotic representation .[ε−1 Q+Q1 ](φ +εφ1 ) = ψ +εvε is resolved by the vectors that are determined by the equality .Π Q1 Π φ = ψ and relations: .

φ1 = R0 (ψ − Q1 φ) vε = Q1 R0 (ψ − Q1 φ).

Here, .R0 = [Q + Π ]−1 − Π is the potential of operator Q. Proposition 1.2 (Singular Perturbation Problem Under the Balance Condition Π Q1 Π = 0) Let Q be a bounded reducible-invertible operator on the Banach space .B with the projector .Π and the potential .R0 . Assume that operators .Q1 and .Q2 are closed with the common domain .B0 and the operator .Π Q0 Π, where −1 . In addition, the .Q0 := Q2 − Q1 R0 Q1 has the inverse operator .(Π Q0 Π ) operator .Q1 satisfies the balance condition .Π Q1 Π φ = 0, φ ∈ B0 . Then the asymptotic representation .

[ε−2 Q + ε−1 Q1 + Q2 ](φ + εφ1 + ε2 φ2 ) = ψ + εvε

.

1.7 Reducible-Invertible Operators and Their Perturbations

17

can be resolved by the vectors that are determined by the equality Π Q0 Π φ = Π ψ

.

and the relations φ1 = −R0 Q1 φ, . φ2 = R0 (ψ − Q0 φ), vε = [Q1 + εQ2 ]φ2 + Q2 φ1 . We note that these two propositions, Propositions 1.1 and 1.2, can be used later to prove averaging/merging and diffusion approximation results, respectively, for discrete-time semi-Markov random evolutions. For further reading on this chapter the following references are of interest [29, 35–39, 46, 51, 53, 60–62, 71, 80, 84, 85, 87, 94, 97, 98, 106, 107, 109, 115, 123, 126, 128, 137, 144, 147, 148, 177, 179, 182].

Chapter 2

Discrete-Time Semi-Markov Chains

2.1 Introduction After the works of Pyke [132, 133] on continuous-time semi-Markov processes, a constantly increasing number of works were published. Few years after we have a first paper on discrete-time semi-Markov chains by Anselone [8], and eleven years after we have a chapter in the book of Howard [70] dedicated on the same subject. But while the literature in discrete-time Markov chains theory and applications is huge, there is only a very small number of works in the literature on SMC and most of this is in hidden semi-Markov models for estimation. Discrete-time semi-Markov chains (SMCs) are only recently used in applications, especially, in DNA analysis, image and speech processing, reliability, etc., see [12] and the references therein. These applications have stimulated a research effort in this domain [12, 104, 105, 127]. In this chapter we present the basic stochastic processes concerned in the present book. Namely, we present semi-Markov chains in general state space, definitions, and their basic properties. Starting from Markov renewal chains, then, we construct the semi-Markov chains and associated processes as the Backward recurrence process, the embedded Markov chains, martingales, etc. The Markov renewal equation and theorem are also presented. An important tool concerning the compensating operator is also presented and used to establish some asymptotic results for the semi-Markov chains in the reduced random media. For basic facts on Markov chains, see Appendix A.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0_2

19

20

2 Discrete-Time Semi-Markov Chains

2.2 Semi-Markov Chains It is worth noticing here that the index .k ∈ N is used for the calendar time, while the index .n ∈ N is used for the number of jumps and .t ∈ R+ for the calendar continuous-time.

2.2.1 Definitions Let .(E, E ) be a measurable space including all singleton .σ -algebra. Notice that .N is the set of nonnegative integer numbers, that is, .N := {0, 1, 2, . . .}, and let us also denote .N∗ := {1, 2, . . .}. Definition 2.1 A nonnegative real-valued function .P (x, B), defined on .E × E , is said to be a sub-Markov transition kernel on .(E, E ), if: 1. For any .x ∈ E, .P (x, ·) is a (positive) measure on .(E, E ) such that .P (x, E) ≤ 1. 2. For any .B ∈ E , .P (·, B) is a .(E , B)-measurable function. In case where .P (x, E) = 1 for any .x ∈ E, the kernel P is said to be a Markov transition kernel or Markov transition function. In case where .P (x, ·) is a signed measure, it will be called a signed transition kernel. For further information on Markov chains, see Appendix A and the references therein. Definition 2.2 A nonnegative real-valued function .q(x, B, k), defined on .E × E × N, is said to be a semi-Markov kernel on .(E, E ), if: 1. For any .x ∈ E, and .B ∈ E , .q(x, B, ·) is a measure on .N. 2. For any .k ∈ N, .q(·, ·, k) is a sub-Markov kernel on .(E, E ). 3. .P (·, ·) := s≥0 q(·, ·, s) is a sub-Markov kernel on .(E, E ). Let .P(N) denote the power set of .N. The following properties follow directly by the above definitions. • For any .x ∈ E, .q(x, ·, ·) defines a probability measure on .(E × N, E ⊗ P(N)). • For any .x ∈ E, the function .fx (·) := q(x, E, ·) is a probability measure on .N. • For any .B ∈ E , .k ∈ N, .q(·, B, k) is a .E -measurable function. Let .(Ω, F , (Fn , n ∈ N), P) be a stochastic basis (a filtered space) on which we consider an adapted double sequence of random variables .(xn , τn , .n ∈ N), where .(xn , n ≥ 0) are E-valued and .(τn , n ≥ 0) are .N-valued with .0 ≤ τ0 < τ1 < . . . < τn < τn+1 < . . . the jump times.

2.2 Semi-Markov Chains

21

Definition 2.3 A sequence of random variables .(xn , τn , n ∈ N) is said to be a Markov renewal process in discrete-time .k ∈ N, with state space .(E, E ), if P(xn+1 ∈ B, τn+1 − τn = k | Fn ) = P(xn+1 ∈ B, τn+1 − τn = k | xn ),

.

a.s.

for any .n ∈ N, .B ∈ E , .k ∈ N∗ , and .τ0 = 0. In the sequel, we will call such a process a Markov renewal chain (MRC). The semi-Markov kernel q is defined by (see, e.g., [12]) q(x, B, k) := P(xn+1 ∈ B, τn+1 − τn = k | xn = x),

.

x ∈ E, B ∈ E , k, n ∈ N. (2.1)

 We will denote also .q(x, B, Γ ) = k∈Γ q(x, B, k), where .Γ ⊂ N. Let us define also the cumulative semi-Markov kernel Q as follows: Q(x, B, k) := q(x, B, [0, k]) = P(xn+1 ∈ B, τn+1 − τn ≤ k | xn = x), (2.2)

.

for .x ∈ E, B ∈ E , k, n ∈ N. It is worth noticing that the semi-Markov kernels defined here are independent of n. That means that the SMCs are homogeneous. Let .(θn , n ≥ 0) be the sequenceof successive inter-jump times, that is, .θn := τn − τn−1 , and .θ0 = τ0 . So, .τn = ns=0 θs . We refer to Markov renewal chain (or process) for both sequences .(xn , τn ) and .(xn , θn ). Define also the sojourn time distributions in a state, .x ∈ E, by .fx (k) := q(x, E, k), and the corresponding cumulative distribution function .Fx (k) := q(x, E, [0, k]), and .F x (k) := 1 − Fx (k) = q(x, E, [k + 1, ∞)). So, we consider that for any .x ∈ E and any .k ∈ N, we have .q(x, {x}, k) = 0, which we call a standard semi-Markov kernel, and any semi-Markov kernel can be transformed to such one. The process .(xn ) is called the Embedded Markov Chain (EMC) of the MRC .(xn , τn ) with transition kernel .P (x, dy). Then the semi-Markov kernel q has the following representation: q(x, dy, k) = P (x, dy)fxy (k),

.

(2.3)

where .fxy (k) := P(τn+1 − τn = k | xn = x, xn+1 = y), the conditional law of the sojourn time in state x given that the next visited state is y. Let us denote the unconditional law of the sojourn time in state x by .fx (k) := P(τn+1 − τn = k | xn = x). In the particular case where .fxy does not depend on the arrival state y, we have .fxy ≡ fx . A more general definition of the Markov renewal chain, in the case where .τ0 ≥ 0, is as follows: Let us consider a Markov transition kernel .P = (P ((x, s); B × Γ ) : (x, s) ∈ E × N, B ∈ E , Γ ∈ P(N)), on .(E × N, E × P(N)). Define now the semi-Markov kernel by q(x, B, k − s) := P ((x, s), B × {k}).

.

22

2 Discrete-Time Semi-Markov Chains

Then, for any .(x, s) ∈ E × N, there exists a probability measure, say .P(x,s) , on (Ω, F ), such that

.

P(x,s) (x0 = x, τ0 = s) = 1,

.

and P(x,s) (xn+1 ∈ B, τn+1 = k | σ (xr , τr ; r ≤ n))

.

= P(x,s) (xn+1 ∈ B, τn+1 = k | σ (xn , τn )) = q(x, B, k − τn ),

P(x,s) − a.s.

Then the coupled process .(xn , τn , n ∈ N) is a Markov renewal chain. For a semi-Markov kernel q on the state space .(E, E ), we can define a standard semi-Markov kernel .q, ˜ defined on the state space .(E × E, E × E ), as follows: .

q((x, ˜ y), A × B, k) = 1A (y)P (y, B)fxy (k).

(2.4)

This semi-Markov kernel is of the abovementioned particular case where the sojourn time in state .(x, y) does not depend on the next visited state. We can see easily that for an MRC .(xn , τn , .n ∈ N), with semi-Markov kernel q, the process .(xn , xn+1 , τn+1 , .n ∈ N), is an MRC with semi-Markov kernel .q. ˜ For the last case, the semi-Markov kernel is q((x, ˜ y), A × B, k) = P˜ ((x, y), A × B)f˜(x,y) (k) = 1A (y)q(y, B, k).

.

where .

f˜(x,y) (k) = fxy (k), P˜ ((x, y), A × B) = 1A (y)P (y, B).

It is worth noticing that following the above analysis there is no restriction in generality to consider, instead of (2.3), semi-Markov kernel of type q(x, dy, k) = P (x, dy)fx (k).

.

(2.5)

Define now the counting process .(νk , k ∈ N) of jumps, by .νk = max{n : τn ≤ k}, and define the discrete-time semi-Markov process .(zk , k ∈ N) by zk = xνk ,

.

for k ∈ N.

In the sequel we will call such a process a semi-Markov chain.

(2.6)

2.2 Semi-Markov Chains

23

The initial probability law .α is defined in the usual way, that is, .α(B) := P(z0 ∈ B) = P(x0 ∈ B), for any .B ∈ E . A Markov renewal chain and a semi-Markov chain are defined by the initial probability .α and the semi-Markov kernel q. Define now the backward recurrence time process .γk := k − τνk , .k ≥ 0 .(τ0 = 0) and the filtration .Fk := σ (zs , γs ; s ≤ k), .k ≥ 0. The coupled process .zk , γk , .k ≥ 0, is a Markov chain with transition kernel .P on .(E × N, E × σ (N)), defined by P ((x, k); B × {s}) =

.

q(x, B, k + 1) F x (k)

1{s=0} +

F x (k + 1) F x (k)

1{s=k+1,s =0} 1B (x) (2.7)

for .F x (k) > 0. The corresponding transition probability operator of .P , also denoted by the same symbol .P on a separable Banach space .B, is defined by P ϕ(x, k) =

.





1 F x (k)

q(x, dy, k + 1)ϕ(y, 0) + E

F x (k + 1) F x (k)

ϕ(x, k + 1), (2.8)

for .(x, k) ∈ E × N. Let us define also the exit “rate” of the SMC from the state .x ∈ E and time .k ∈ N, given by λx (k) := Px (τ1 = k | τ1 ≥ k).

.

Of course, as is well known, the transition rate in discrete-time is a probability and not a positive real-valued function as is the case in continuous-time. Now the above relation (2.8) can be written also in the following interesting form: P ϕ(x, k) = ϕ(x, k + 1) + λx (k + 1)[P ϕ(x, 0) − ϕ(x, k + 1)].

.

(2.9)

The above relation is similar to the generator of the process .(zt , γt ) in the continuous-time, see, Relation (1.25), Chap. 1, and, e.g., [90–92]. The stationary distribution of the process .(zk , γk ), if it exists, is given by π (dx × {k}) = ρ(dx)F x (k)/m,

.

(2.10)

where  m :=

ρ(dx)m(x),

.

E

(2.11)

24

2 Discrete-Time Semi-Markov Chains

and .m(x) is the mean sojourn time in state .x ∈ E, that is, m(x) =



.

(2.12)

F x (k),

k≥0

and .ρ(dx) is the stationary distribution of the EMC .(xn ). The probability measure .π defined by π(B) = π (B × N) =

.

1 m

 ρ(dx)m(x)

(2.13)

B

is the stationary probability of the SMC .(zk ). Define also the r-th moment of holding time in state .x ∈ E, mr (x) :=



.

k r q(x, E, k),

r = 1, 2, . . .

k≥1

Of course, .m(x) = m1 (x), for any .x ∈ E. Finally, it is straightforward to see that .π P = π . Define now the uniform integrability of the r-th moments of the sojourn time in states by .

lim sup

M→∞ x∈E



k r fx (k) = 0,

(2.14)

k≥M

for any .r ≥ 1. It is worth noticing that the stationary probability .π of the SMC .(zk ) exists if the EMC .(xn ), with transition kernel P , is ergodic with ergodic probability .ρ and .0 < m < ∞. The same conditions imply also the ergodicity of the SMC as we will see below in the application of Markov renewal theorem for the transition function. The following are some examples of SMC. Example 2.1 (Markov chain) A Markov chain is a particular case of a semi-Markov chain. Consider a discrete state space, i.e., .E = N, and transition probabilities .(pij ). Then the semi-Markov kernel q, of this Markov chain, can be written q(i, {j }, k) = pij piik−1 ,

.

k ≥ 1,

with .q(i, j, 0) = 0, and we put .00 = 1. In fact, the embedded Markov chain has transition probabilities .P (i, j ) = pij /(1 − pii ), .(j = i), and the sojourn time distribution in the state i is .fi (k) = (1 − pii )piik−1 (the geometric distribution with parameter .1 − pii ). The function .λx (k), in the case of Markov chain, is constant, with respect to k, denoted .λi , .i ∈ E, and we have .λi = 1 − pii .

2.2 Semi-Markov Chains

25

Example 2.2 (Alternating renewal chain) Let us consider an alternating renewal process with up times: .X1 , X2 , . . ., i.i.d., and down times: .Y1 , Y2 , . . ., i.i.d., and the two sequences are independent between them. The common laws of up and down times are, respectively, f and g on .N. Denote now by .Sn the arrival time of the .(n + 1)-th cycle: τn =

.

n  (Xi + Yi ),

n ≥ 1,

τ0 = 0.

i=1

The process zk =



.

1{τn ≤k 0, and semi-Markov kernel  1 |x| |x| .q(x, B, k) := (a − |y|)dy [p(k−1) − pk ]1−δ(x) , 2a B where B is any measurable subset of .[−a, a], .x ∈ [−a, a], .k ∈ N, .0 < p < 1 and δ(x) = 1 for .x = 0 and .δ(x) = 0 for .x = 0.

.

Example 2.4 (Semi-Markov random walk on the half-line) Let us consider a distribution function, say F , on .(R, B), and a sequence of r.v. .(ξn , n ≥ 0), i.i.d., with common distribution F , .F (B) = P(ξ ∈ B), .B ∈ B. Define a Markov transition kernel by  P (x, B) =

.

F (B − x), f or B a measurable subset of (0, ∞) F (−∞, −x], for B = {0}.

Define now for each .x ∈ R+ the discrete-time Weibull distribution, .W (qx , bx ), (k−1)bx − where .0 < qx < 1, and .bx > 0. Then the kernel .q(x, B, k) = P (x, B)[qx b x qxk ], .k ≥ 1, is a semi-Markov kernel.

26

2 Discrete-Time Semi-Markov Chains

Define now the stationary projection operator .Π on the null space .N(Q ) of the discrete generating operator .Q := P − I , Π ϕ(x, s) =



.

π (dy × {s})ϕ(y, s)1(x, s),

s≥0 E

where .1(x, s) = 1 for any .x ∈ E and .s ∈ N. This operator satisfies the equalities Π Q = Q Π = 0.

.

The potential operator of .Q , denoted by .R0 , is defined by R0 := (Q + Π )−1 − Π =



.

[(P )k − Π ].

k≥0

See Sect. 1.7 in Chap. 1.

2.2.2 Classification of States Let us consider a semi-Markov chain .(zk , k ∈ N), with state space .(E, E ). Consider also a non-empty set .B ∈ E and the random variable τB := inf{k > 0 : zk ∈ B},

.

(2.15)

with .inf Ø = ∞. The r.v. .τB is the first hitting time into set B of the semi-Markov chain .(zk ) and it is a stopping time with respect to the filtration .Fk := σ (zs , s ≤ k), .k ∈ N. Denote .Λ(x, B, k) := Px (τB ≤ k), and let .σ be a finite measure on .(E, E ) such that .σ (E) > 0. Definition 2.4 1. A set .B ∈ E is said to be accessible from a state .x ∈ E, if .Λ(x, B, ∞) > 0. 2. The SMC .(zk , k ∈ N) is said to be .σ -irreducible if, whenever .σ (B) > 0, the set B is accessible from any .x ∈ E. 3. An SMC is said to be .σ -recurrent, if whenever .σ (B) > 0, we have .Λ(x, B, ∞) = 1, for any .x ∈ E. Theorem 2.1 An SMC .(zk , k ∈ N) is .σ -irreducible (respectively, .σ -recurrent) if and only if the EMC .(xn , n ∈ N) is .σ -irreducible (respectively, .σ -recurrent). Definition 2.5 A set .B ∈ E is said to be recurrent (respectively, transient) for the SMC .(zk , k ∈ N) if it is recurrent (respectively, transient) for the EMC .(xn , n ∈ N), that is, .Px (xn ∈ B, i.o.) = 1, for any .x ∈ E.

2.2 Semi-Markov Chains

27

2.2.3 Markov Renewal Equation and Theorem Consider a measurable function .ϕ : E × N → R, and define its convolution by q as follows: q ∗ ϕ(x, k) =

k  

.

q(x, dy, s)ϕ(y, k − s).

(2.16)

s=0 E

Following (2.16), define now the n-fold convolution of q by itself, as follows: q

.

(n)

(x, B, k) =

k  

q(x, dy, s)q (n−1) (y, B, k − s),

n ≥ 1,

s=0 E

x ∈ E, B ∈ E , k ∈ N, and .q (0) (x, B, k) = 1B (x)δ(k). Here .δ is the Dirac distribution at 0, that is, .δ(k) := δ({k}) = 1 if .k = 0 and .δ(k) = 0 if .k = 0. So, the element .q (0) is the (left) identity for the convolution product, that is, .q (0) ∗ϕ = ϕ, and .q (0) ∗q (n) = q (n) ∗q (0) = q (n) . Consequently .q (1) (x, B, k) = q(x, B, k). Moreover, we have that .q (n) (x, B, k) = 0, for .n > k. Define also the Markov renewal function .ψ by ψ(x, B, k) :=

k 

.

q (n) (x, B, k).

(2.17)

n=0

Let us consider two measurable functions .u, v : E × N → R. The Markov renewal equation, with .v a given function and u the unknown one, is u(x, k) = v(x, k) +

k  

.

s=0

q(x, dy, s)u(y, k − s)

E

or, equivalently, u(x, k) = v(x, k) + q ∗ u(x, k)

.

or (q (0) − q) ∗ u = v.

.

(2.18)

28

2 Discrete-Time Semi-Markov Chains

For example, the Markov Renewal function (2.17) can be written in the following form which is a particular Markov renewal equation ψ(x, B, k) = q (0) (x, B, k) +

k  

q(x, dy, s)ψ(y, B, k − s).

.

s=0 E

This equation can also be written as (q (0) − q) ∗ ψ = q (0) ,

.

and then we get ψ = (q (0) − q)(−1) .

.

Here .a (−1) means inversion in the convolution sense, that is, .a (−1) ∗ a = q (0) . The following theorem is a discrete-time version of the Markov renewal theorem given by Shurenkov in [143]. Let us consider the following assumptions: A1: The Markov chain .(xn ) is ergodic with stationary distribution .ρ(B), B ∈ E . A2: The m, given in (2.11), if finite and positive, that is, .0 < m < ∞. Theorem 2.2 (Markov Renewal Theorem) Under Assumptions A1–A2 and considering that the function .v(x, k) is such that  ρ(dx)

.

E



|v(x, k)| < ∞,

k≥0

equation (2.18) has a unique solution given by .u(x, k) = ψ ∗ v(x, k), and .

lim ψ ∗ v(x, k) =

k→∞

1 m

 ρ(dx) E



v(x, k).

k≥0

Interesting applications of the above Markov renewal theorem are given in the following examples. Example 2.5 The transition function of a semi-Markov chain (.zk ) is defined by Pk (x, B) := P(zk ∈ B | z0 = x),

.

x ∈ E, B ∈ E ,

where we suppose .τ0 = 0. This function fulfils the following MRE: Pk (x, B) = 1B (x)F x (k) +

k  

.

s=0 E

q(x, dy, s)Pk−s (y, B).

2.2 Semi-Markov Chains

29

From the Markov renewal theorem, under Assumptions A1–A2, we get the solution Pk (x, B) = ψ ∗ 1B (x)F x (k) =

k  

ψ(x, dy, s)F y (k − s)

.

(2.19)

s=0 B

and the stationary probability .π of .(zk ), as the following limit: .

lim Pk (x, B) =

k→∞

1 m

 ρ(dx)m(x) =: π(B). B

The above probability .π(B) is the stationary probability of the SMC. Thus we have 

1 .π(B) = m

ρ(dx)m(x) = π (B × N).

(2.20)

B

Compare with (2.13). It is worth noticing here that we do not have .π Pk = π as in the Markov case. Example 2.6 For the alternating renewal sequence, see Example 2.2, we obtain its transition function in the matrix form   F f ∗G (k), .P (k) = M ∗ g∗F G where M is the renewal function of the inter-arrival time law .h := f ∗ g, that is, (see, e.g., [12]) M(k) =

k 

.

h(s) (k)

s=0

and F and G are the corresponding cumulative distribution functions: f and g, respectively. In fact, .h(s) corresponds to the convolution of semi-Markov kernel (2.16) when the state space E includes just one element. The asymptotic result by application of the Markov renewal theorem gives the following result:   m1 m0 /(m1 + m0 ), . lim P (k) = k→∞ m1 m0 where .m1 and .m0 are the mean sojourn times on states 1 and 0, respectively, that is, the means of the laws f and g, respectively. And finally, the stationary probability, .π = (π1 , π0 ), of the semi-Markov chain .(zk ) is π1 =

.

m1 m1 + m0

and

π0 =

m0 . m1 + m0

30

2 Discrete-Time Semi-Markov Chains

Example 2.7 Let us consider a stochastic system whose temporal behavior is described by a semi-Markov chain .(zk ) with state space .(E, E ), semi-Markov kernel q, and the initial probability .α, that is, .α(B) := P(z0 ∈ B), .B ∈ E . Consider the following partition of the state space .E := E0 ∪ {0}, where .E0 is the transient states set and 0 is an absorbing state. Let us define the hitting time .τ to state 0, that is, τ := inf{k ≥ 0 : zk = 0},

.

(inf Ø = +∞),

and the conditional hitting time survival function .Sx , by Sx (k) := Px (τ > k) = Px (zs ∈ E0 , ∀s ≤ k),

.

x ∈ E0 .

Of course, .Sx (k) = 0 for .x = 0. Let us define the Markov renewal function .ψ0 , ψ0 (x, B, k) :=

k 

.

(n)

q0 (x, B, k),

n=0

where .q0 is the restriction of the semi-Markov kernel q on .E0 × E0 , and .x ∈ E, B ∈ E0 , k ∈ N. Proposition 2.1 The hitting time survival function .Sx (k) satisfies the MRE: Sx (k) = F x (k) +

k  

q(x, dy, s)Sy (k − s),

.

x ∈ E0 .

(2.21)

s=0 E0

Hence, the hitting time survival function is given by the unique solution of the above equation by Sx (k) = (ψ0 ∗ F · 1E0 )(x, k),

.

where .F · 1E0 (x, k) = 1E0 (x)F x (k). Finally the (unconditional) survival function is given by S(k) =

k  

 α(dx)ψ0 (x, dy, s)F y (k − s).

.

s=0 E0

(2.22)

E0

Remark 2.1 The above formula (2.22) is a generalization to the SMC of the phase type distribution functions in the discrete-time Markov chain case (see, e.g., [119]).

2.4 Compensating Operator and Martingales

31

2.3 Discrete- and Continuous-Time Connection The measurable space .(E, E ) will be here the common state space of the semiMarkov process and SMC. Let us consider a continuous-time semi-Markov process .(z(t), t ∈ R+ ) with semi-Markov kernel .Q(x, B, t), .x ∈ E, .B ∈ E , and .t ∈ R+ , see, e.g., [104]. We can consider a discrete-time SMC by discretization of the time as follows: Let .h > 0 be a fixed constant, the unit of time, and consider the time points on .R+ , .tk = kh, .k ∈ N. Define q(x, B, k) := Q(x, B, kh) − Q(x, B, (k − 1)h),

.

k ≥ 1,

q(0) = 0.

Then q is a semi-Markov kernel which defines a semi-Markov chain, say .z˜ k . This SMC is an approximation of the continuous-time semi-Markov process, both, of course, with the same state space. It is worth noticing that a more general discretization can be performed, instead of the regular one used here. Now we can consider the opposite. For an SMC, with semi-Markov kernel q, we can consider a semi-Markov process in continuous-time with semi-Markov kernel Q, as follows: Q(x, B, t) :=



.

q(x, B, s)1{kh≤t 0, for all .i = 1, 2, . . . , d. Let us also consider the trace of .σ -algebra .E on .Ej , denoted by .Ej , for .j = 1, . . . , d.  := {1, 2, . . . , d} defined by Consider also the merging function .υ : E → E υ(x) = i,

.

if x ∈ Ei .

 and semiLet us also consider the MRC .( xn ,  θn , n ≥ 0), with state space .E,  Markov kernel . q , that is, . qij (k) :=  q (i, {j }, k) = P( xn+1 = j, θn+1 = k |  xn = i), given by  . qij (k) := ρ(dx)q(x, Ej , k)/ρ(Ei ). (2.29) Ei

And, from (2.29), we obtain  .

p ij := P( xn+1 = j |  xn = i) =

ρ(dx)P (x, Ej )/ρ(Ei ) Ei

and x (k) := P( .f θn+1 = k |  xn = i) =

 ρ(dx)fx (k)/ρ(Ei ). Ei

Define also the moments when the EMC .(xn ) is observed in the set .Ei (i) (i) νm := inf{n > νm−1 : xn ∈ Ei },

.

ν0(i) = 0,

(2.30)

for any .i = 1, 2, . . . , d. The following result is a discrete-time version of the Arjas–Koroliuk Theorem [9]. Theorem 2.4 Suppose that the EMC .(xn ) is ergodic, with ergodic probability .ρ, such that .ρ(Ei ) > 0, for .1 ≤ i ≤ d, and then we have the following convergence: .

lim P( xν (i) ∈ Ej ,  θν (i) = k) =  qij (k),

m→∞

m+1

m+1

for any .i, j = 1, 2, . . . , d. Moreover, we get in a straightforward way from the above theorem that .

lim P( xν (i) ∈ Ej ) = p ij ,

m→∞

m+1

 i, j ∈ E

36

2 Discrete-Time Semi-Markov Chains

and .

lim P( θν (i) = k) = fi (k),

m→∞

 i ∈ E.

m+1

For discrete state space SMC, see, e.g., [12].

2.6 Semi-Markov Chains in Merging State Space Merging an SMC is the asymptotic aggregation of state space to a simpler state space, here this will be a finite state space. The resulting limit process on the finite state space is a Markov chain in continuous-time. We consider two cases, the ergodic and non-ergodic with an absorbing state in the last case. The split sets .Ej are merged in one state each one.

2.6.1 The Ergodic Case Let us consider a family of ergodic semi-Markov chains .(zkε , k ≥ 0, ε > 0), with semi-Markov kernels .(q ε , ε > 0) and a fixed state space .(E, E ), a measurable space. Let us consider the following partition (split) of the state space: E = ∪dj =1 Ej ,

.

Ei ∩ Ej = Ø,

i = j

(2.31)

and the trace of .σ -algebra .E on .Ej , denoted by .Ej , for .j = 1, . . . , d. We assume that the semi-Markov kernels have the following representation: q ε (x, B, k) = P ε (x, B)fx (k),

.

(2.32)

where the transition kernel of the EMC .(xnε , n ≥ 0) has the representation P ε (x, B) = P (x, B) + εP1 (x, B),

.

(2.33)

where P is a Markov transition kernel and .P1 is a signed transition kernel on .(E, E ). The transition kernel P determines a support Markov chain, say .xn0 , .n ≥ 0, and satisfies the following relations:  P (x, Ej ) = 1j (x) ≡ 1Ej (x) =

.

1 if x ∈ Ej 0 if x ∈

Ej ,

(2.34)

for .j = 1, . . . , d. Of course, the signed perturbing kernel .P1 satisfies the relation P1 (x, E) = 0, and .P ε (x, E) = P (x, E) = 1.

.

2.6 Semi-Markov Chains in Merging State Space

37

The perturbing signed transition kernel, .P1 , provides transition probabilities between merged states.  be the merging onto function defined by .υ(j ) = s, if .j ∈ Es , Let .υ : E → E  .s ∈ E = {1, . . . , d}. Set .k := [t/ε], where .[x] is the integer part of the positive real number x, and define the split family of processes ε  xtε := υ(z[t/ε] ),

.

t ≥ 0, ε > 0.

(2.35)

Define also the projector operator .Π onto the null space, .N(Q), of the operator Q := P − I , by

.

 Π ϕ(x) =  ϕ (υ(x)),

.

where  ϕ (j ) :=

(2.36)

ρj (dx)ϕ(x). Ej

This operator satisfies the equations (see Sect. 1.7, Chap. 1) Π Q = QΠ = 0.

.

The potential operator of Q, denoted by .R0 , is defined by R0 := (Q + Π )−1 − Π =



.

[P k − Π ].

k≥0

Let us now consider the following assumptions needed in the sequel. C1: The transition kernel .P ε (x, B) of the embedded Markov chain .(xnε ) has the representation (2.33). C2: The supporting Markov chain .(xn0 ) with transition kernel P is uniformly  that is, ergodic in each class .Ej , with stationary distribution .ρj (dx), .j ∈ E,  ρj (B) =

ρj (dx)P (x, B),

.

and

ρj (Ej ) = 1,

B ∈ Ej .

Ej

C3: The average exit probabilities of the initial embedded Markov chain .(xnε ) are positive, that is,  .

p j :=

ρj (dx)P1 (x, E \ Ej ) > 0. Ej

C4: The mean merged values are positive and bounded, that is,  0 < mj :=

ρj (dx)m(x) < ∞.

.

Ej

38

2 Discrete-Time Semi-Markov Chains

From relation (2.20), we obtain directly πj (dx)q(x) = qj ρj (dx),

(2.37)

.

where .q(x) := 1/m(x) and .qj := 1/mj with .mj := Ej ρj (dx)m(x). The following results express the asymptotic merging of the initial SMC to a  We may call this result Markov process in continuous-time and finite state space .E. also an asymptotic aggregation or lumping of SMC. This kind of results is important in many extends, theoretical and practical. Theorem 2.5 Under Assumptions C1–C4, the following weak convergence takes place:  xtε ⇒  xt

.

as ε → 0,

(2.38)

where the limit merged process . xt is a continuous-time Markov process determined  = {1, . . . , d}, by the intensity matrix on the state space .E   = ( Q qij ; i, j ∈ E),

.

where   qij =

.

and .p ij :=

Ei

qi p ij , j = i ii , j = i, −qi p

 and .qi := ρi (dx)P1 (x, Ej ), with .i, j ∈ E,

Ei

πi (dx)q(x).

Proof Let us consider the extended Markov renewal process xnε ,

.

υ(xnε ),

τnε ,

t ≥ 0, ε > 0,

(2.39)

ε ε ]. where .n := [t/ε], .xnε = zε (τnε ) and .τn+1 = τnε + [εθn+1 The extended compensating operator of this process is defined by the following relation (see Sect. 2.4 and [90]):

 Lε ϕ(x, v(x), k) = ε−1 q(x) E[ϕ(x1ε , υ(x1ε ), τ1ε ) | x0ε = x, υ(x0ε ) = j, τ0ε = k] −ϕ(x, j, k) . (2.40)

.

The extended compensating operator .Lε acting on test functions .ϕ(x, υ(x)), .x ∈ E, can be written as  ε −1 .L ϕ(x, υ(x)) = ε q(x) P ε (x, dy)[ϕ(y, υ(y)) − ϕ(x, υ(x))]. (2.41) E

2.6 Semi-Markov Chains in Merging State Space

39

And now from (2.41), the operator .Lε can be written as follows: .

Lε = ε−1 Q + Q1 ,

(2.42)

P (x, dy)[ϕ(y, υ(y)) − ϕ(x, υ(x))],

(2.43)

where  Qϕ(x, υ(x)) = q(x)

.

E

and  Q1 ϕ(x, υ(x)) = q(x)

P1 (x, dy)ϕ(y, υ(y)).

.

(2.44)

E

Now, by the following singular perturbation problem, on test functions ϕ ε (x, υ(x)) =  ϕ (υ(x)) + εϕ1 (x, υ(x)),

.

Lε ϕ ε (x, υ(x)) = Lϕ(υ(x)) + εθ ε (x),

(2.45)

.

and from Proposition 1.1, in Chap. 1, (see also Proposition 5.1 in [90]), we get the L, is defined by the relation limit operator .L, whose contracting form, . Π Q1 Π =  LΠ

.

≡ provides us directly the generator .Q xt , and the proof is L of the limit process . achieved. .

2.6.2 The Non-ergodic Case Let us consider a family of semi-Markov chains .(zkε , k ≥ 0, ε > 0), with semiMarkov kernels .q ε and a fixed state space .(E  , E  ), a measurable space, which includes an absorbing state, say 0. Of course, in case where we have a final class, we can replace it by a state 0, and the analysis presented here remains the same. Let us consider the following partition of the state space. E  = E ∪ {0},

.

E = ∪dj =1 Ej ,

Ei ∩ Ej = Ø,

i = j.

(2.46)

0 be the merging onto function defined by .υ(j ) = s, if .j ∈ Es , Let .υ : E  → E  .s ∈ E0 = {0, 1, . . . , d}, and .υ(0) = 0.

40

2 Discrete-Time Semi-Markov Chains

We now need the following additional condition. C5: The average transition probabilities of the initial embedded Markov chain .(xnε ) to state 0, satisfy the relation,  .

p j 0 := −

ρj (dx)P1 (x, E) > 0 Ej

for .j = 1, . . . , d. ε Let us also define the absorption time of the initial process .z[t/ε] to the state 0, for any .ε > 0,

ε .ζ ,

ε ζ ε := inf{t ≥ 0 : z[t/ε] = 0}.

.

(2.47)

Theorem 2.6 Under Assumptions C1-C4 and C5, the following weak convergence takes place  xtε ⇒  xt

.

as ε → 0,

(2.48)

where the limit merged process . xt , .0 ≤ t ≤  ζ , is a continuous-time Markov process  determined, on the state space .E = {0, 1, . . . , d}, by the intensity matrix 0 ),  = ( Q qij ; i, j ∈ E

.

where ⎧ ij , j = i, i = 0 ⎨ qi p . qij = −qi p ii , j = i, i = 0 ⎩ 0, i=0 x (t) = 0}. and  .ζ := inf{t ≥ 0 :  It is worth noticing here that the distribution of  .ζ is a phase type in continuoustime, see, e.g., [119], that is, ˆ

Pα (ζˆ > t) = αeQt 1

.

and .Eα ζˆ = −αQ−1 1, where .α is any initial distribution on .Eˆ \ {0}, and .1 = (1, 1, . . . , 1) a d-dimensional column vector. ˆ we could It is worth noticing that instead of a finite merged state space .E, ˆ Eˆ ), in both cases. The analysis consider a general measurable state space, say .(E, is the same. Nevertheless, if .Eˆ is not compact, we need to complete the proofs with a tightness analysis.

2.7 Concluding Remarks

41

2.7 Concluding Remarks This chapter, dedicated to semi-Markov chains with general state space, describes one of the two key notions of our book: the SMC, which is the switching process for random evolutions, as well as for all stochastic systems considered in sequel. We presented the Markov renewal theorem, the compensating operator, some martingale properties, etc. Moreover, stationary phase merging and merging results of the state space are given. The merging approach can be extended in several ways, e.g., double merging. Also, the results presented here can be extended and used independently in several applied directions. For further reading on this chapter the following references are of interest [10, 11, 15, 42–44, 56, 75–77, 110, 117, 122, 170, 171, 173].

Chapter 3

Discrete-Time Semi-Markov Random Evolutions

3.1 Introduction In this chapter, we define and discuss basic properties of discrete-time semiMarkov random evolutions (DTSMRE). Random evolution is a powerful technique to study stochastic evolutionary systems, see, e.g., [90, 91, 166]. Discrete-time random evolutions, induced by discrete-time Markov chains, are introduced by Cohen [30] and Keepler [82], and discrete-time semi-Markov random evolutions by Limnios [101]. DTSMRE is studied in [104] where asymptotic results are obtained. Swishchuk and Wu [166] and Koroliuk and Limnios [90] studied discrete-time random evolutions induced by the embedded Markov chain of continuous-time semi-Markov processes. This is equivalent to discrete-time Markov random evolution stopped in random time. Discrete-time semi-Markov chains (SMC) are only recently used in applications. Especially, in DNA analysis, image and speech processing, reliability, etc., see [12] and references therein. These applications have stimulated a research effort in this area. But while the literature in discrete-time Markov chains theory and applications is huge, there is only a very small literature on SMC, and most of this is in hidden semi-Markov models for estimation. In this chapter, we present discrete-time random evolutions starting from the Markov case, where especially for the finite state space we give a vector representation, the family of concerned operators, and the Markov renewal equation. Next we present the semi-Markov case where we give the Markov renewal equation of the random evolution. Finally, we present the application to concrete systems, namely, additive functionals, geometric Markov renewal chain, and dynamical systems or difference equations.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0_3

43

44

3 Discrete-Time Semi-Markov Random Evolutions

3.2 Discrete-time Random Evolution with Underlying Markov Chain Consider a separable Banach space .(B, ·) and the space of all bounded linear operators .L(B) on .B. Let .(xn , n ∈ N) be a Markov chain on the finite state space .E := {1, 2, . . . , d} with transition probability matrix .P = (pij , i, j ∈ E). And let .(ϑn , n ≥ 0) be the shift operators on the trajectories of the Markov chain. Define now the operators .D(i), .i ∈ E, in .L(B), and the operators .Φn , .n ∈ N, by Φn = D(xn )D(xn−1 ) · · · D(x0 ),

n≥1

.

and .Φ0 = I (the identity operator on .B). This is a forward random evolution. The backward random evolution is defined by .Ψ0 = I and .Ψn = D(x1 )D(x2 ) · · · D(xn ), .n ≥ 1. As we are going to consider only the forward random evolution, we will say just random evolution. Clearly, .(Φn , n ≥ 0) is adapted to the natural filtration of the Markov chain .(xn ), .Fn := σ (xs , s ≤ n), .n ≥ 0. From the very definition, we have Φn+m = (Φn ◦ ϑm )Φm = (Φm ◦ ϑn )Φn ,

.

for any .n, m ∈ N. This is a multiplicative law. Example 3.1 Let .B = 1 (R) (or .C 0 (R)) and .a : E → R a real-valued function. Define the operators .D(i), .i ∈ E, by D(i)ϕ(u) := ϕ(u + a(i)),

.

and the iterates D k (i)ϕ(u) := ϕ(u + a(i)k),

.

k ≥ 1,

and D 0 (i)ϕ(u) := ϕ(u),

.

then Φn ϕ(u) = ϕ(u +

n 

.

s=1

a(xs )),

n ≥ 1.

3.2 Discrete-time Random Evolution with Underlying Markov Chain

45

 Denote by . B := B ⊕ · · · ⊕ B the direct  sum. An element .ϕ ∈ B is denoted by ϕ := (ϕ1 , . . . , ϕd ), with norm .ϕ := di=1 ϕi , and the direct sum

.

⎞ D(1)ϕ1 ⎟ ..  =⎜ .Dϕ ⎠. ⎝ . ⎛

D(d)ϕd Define now the average random evolution .u(n), by n ϕ)i = Ei [Φn ϕxn ] := E[Φn ϕxn | x0 = i], ui (n) := (Φ

.

i ∈ E.

(3.1)

The following properties of .u(n) are important. n , n ≥ 0) is a contraction discrete Proposition 3.1 The sequence of operators .(Φ  parameter semigroup on .L(B). Proof The contraction is clear from the definition of the operators. Let us prove the semigroup property by using the Markov property and the measurability of .Φn , with respect to .Fn , n+m ϕ)i = Ei [Φn+m ϕxn+m ] (Φ

.

= Ei [Ei (Φn+m ϕxn+m | Fn )] = Ei [Ei ((Φm ◦ ϑn )Φn ϕxm ◦ϑn | Fn )] = Ei [Φn Exn (Φm ϕxn )] = Ei [Φn (Φm ϕxm )] n (Φ m ϕ)]i . = [Φ Proposition 3.2 For any fixed .ϕ ∈  B and .n > 0, we have the Markov renewal type equation ui (n) = (pii )n D n (i)ϕi +

n  

.

pij (pii )k−1 D k (i)uj (n − k).

k=1 j ∈E

Proof By denoting .τ1 the first jump time of the Markov chain .(xn ), we have n ϕ)i = Ei [Φn ϕxn ] = Ei [Φn ϕxn 1(τ1 > n)] + Ei [Φn ϕxn 1(τ1 ≤ n)]. (Φ

.

Now, we have Ei [Φn ϕxn 1(τ1 > n)] = Ei [Φn ϕxn | τ1 > n]Pi (τ1 > n)

.

= (pii )n D n (i)ϕi

46

3 Discrete-Time Semi-Markov Random Evolutions

and Ei [Φn ϕxn 1(τ1 ≤ n)] =

n  

.

Ei [Φn ϕxn 1(τ1 = k, xτ1 = j )]

k=1 j ∈E

=

n  

Ei [Φn ϕxn−k | τ1 = k, xτ1 = j ]Pi (τ1 = k, xτ1 = j )

k=1 j ∈E

=

n  

pij (pii )k−1 D k (i)Ej [Φn−k ϕxn−k ].

k=1 j ∈E



By integrating the last two parts, the proof is completed.

.

In a similar way, we can handle a jump operator, say .Γ (i, j ), .i, j ∈ E by considering the random evolution Φn = D(xn )Γ (xn−1 , xn )D(xn−1 ) · · · D(x1 )Γ (x0 , x1 ),

.

n ≥ 1,

and .Φ0 = I .

3.3 Definition and Properties of DTSMRE Let .(E, E ) be a measurable space of which .σ -algebra .E includes all singleton .{x}, x ∈ E. Consider also a stochastic basis .(Ω, F , (Fn )n∈N , P) on which we consider a Markov renewal chain .(xn , τn , n ∈ N) in discrete time .k ∈ N, with state space .(E, E ) and semi-Markov kernel q, that is, .

q(x, B, k) := P(xn+1 ∈ B, τn+1 − τn = k | xn = x),

.

x ∈ E, B ∈ E , k, n ∈ N. (3.2)

See Chap. 2, and also, in [12, 101].  We will also denote .q(x, B, Γ ) = n∈Γ q(x, B, n), where .Γ ⊂ N. The process .(xn ) is the embedded Markov chain of the MRC .(xn , τn ) with transition kernel .P (x, dy). The semi-Markov kernel q is written as q(x, dy, k) = P (x, dy)fx (k),

.

where .fx (k) := P(τn+1 − τn = k | xn = x), the law of the sojourn time in state x. Define also the counting process of jumps .νk = max{n : τn ≤ k}, and the discrete-time semi-Markov chain .(zk ) by .zk = xνk , for .k ∈ N. Define now the backward recurrence time process .γk := k − τνk , .k ≥ 0, and the filtration .Fk := σ (zs , γs ; s ≤ k), .k ≥ 0.

3.3 Definition and Properties of DTSMRE

47

Let us consider a separable Banach space .B of real-valued measurable functions defined on .E × N, endowed with the sup norm .· and denote by .B its Borel .σ algebra. The Markov chain .(zk , γk , k ≥ 0), has the following transition probability operator: .P on B P ϕ(x, k) =

.



1

q(x, dy, k + 1)ϕ(y, 0) +

F x (k)

F x (k + 1) F x (k)

E\{x}

ϕ(x, k + 1), (3.3)

and its stationary distribution, if there exist, is given by π (dx × {k}) = ρ(dx)F x (k)/m,

.

where  m :=

ρ(dx)m(x),

.

m(x) =

E



F x (k),

k≥0

and .ρ(dx) is the stationary distribution of the EMC .(xn ), .Fx (k) := q(x, E, [0, k]), and .F x (k) := 1 − Fx (k) = q(x, E, [k + 1, ∞)). The probability measure .π defined by .π(B) = π (B × N) is the stationary probability of the SMC .(zk ). Define also the r-th moment of the holding time in state .x ∈ E, mr (x) :=



.

k r q(x, E, k),

r = 1, 2, . . .

k≥1

Of course, .m(x) = m1 (x), for any .x ∈ E. Define now the stationary projection operator .Π on the null space of the (discrete) generating operator .Q := P − I , Π ϕ(x, s) =



.

π (dy × {s})ϕ(y, s)1(x, s),

s≥0 E

where .1(x, s) = 1 for any .x ∈ E, and .s ∈ N. This operator satisfies the equations Π Q = Q Π = 0.

.

The potential operator of .Q , denoted by .R0 , is defined by R0 := (Q + Π )−1 − Π =



.

k≥0

See Sect. 1.7, Chap. 1.

[(P )k − Π ].

48

3 Discrete-Time Semi-Markov Random Evolutions

Let us give a family of bounded contraction operators .D(x), x ∈ E, defined on .B, where the maps .D(x)ϕ : E → B are .E -measurable, .ϕ ∈ B. Denote by I the identity operator on .B. Let .Π B = N(Q ) be the null space, and .(I − Π )B = R(Q ) be here that the Markov chain the range values space of operator .Q . We will suppose

n .(zk , γk , k ∈ N) is uniformly ergodic, that is, . ((P ) − Π )ϕ → 0, as .n → ∞, for any .ϕ ∈ B. In that case, the transition operator is reducible invertible on B. Thus, we have .B = N (Q ) ⊕ R(Q ), the direct sum of the two subspaces. The domain of an operator A on .B is denoted by .D(A) := {ϕ ∈ B : Aϕ ∈ B}. Let us define now a discrete-time semi-Markov random evolution (DTSMRE). Definition 3.1 A (forward) discrete-time semi-Markov random evolution .(Φk , k ∈ N), on .B, is defined by Φk ϕ = D(zk )D(zk−1 ) · · · D(z2 )D(z1 )ϕ,

.

k ≥ 1,

and

Φ0 = I,

(3.4)

for any .ϕ ∈ B0 := ∩x∈E D(D(x)). Thus we have .Φk = D(zk )Φk−1 . Define also the filtration .Fk := σ (zs , γs ; s ≤ k), .k ≥ 0. Example 3.2 Consider an additive functional of the SMC .(zk , k ∈ N), that is, αk := u +

k 

.

a(zs ),

for

k ≥ 1,

with α0 = u.

s=1

Define now a family of operators .D(x), x ∈ E, on .B, by D(x)ϕ(u) = ϕ(u + a(x)).

.

Then we can write Φk ϕ(u) = D(zk ) · · · D(z1 )ϕ(u) = ϕ(u +

k 

.

a(zs )) = ϕ(zk ).

s=1

It is worth noticing that operators .D(x), x ∈ E, are commutative, and then we can write Φk ϕ(u) =

k

.

D(zs )ϕ(u) = ϕ(zk ).

s=1

The process .Mk defined by Mk := Φk − I −

k−1 

.

s=0

E[Φs+1 − Φs | Fs ],

k ≥ 1,

M0 = 0,

(3.5)

3.3 Definition and Properties of DTSMRE

49

on .B, is an .Fk -martingale. The random evolution .Φk can be written as follows: Φk := I +

.

k−1  [D(zs+1 ) − I ]Φs , s=0

and then the martingale (3.5) can also be written as follows: Mk := Φk − I −

k−1 

.

E[(D(zs+1 ) − I )Φs | Fs ],

s=0

or Mk := Φk − I −

k−1 

.

[E(D(zs+1 ) | Fs ) − I ]Φs .

s=0

Finally, as E[D(zs+1 )Φs ϕ | Fs ] = (P D(·)Φs ϕ)(zs , γs ),

.

one takes Mk := Φk − I −

k−1 

.

[P D(·) − I ]Φs .

s=0

Let us define now the average random evolution .uk (x), .x ∈ E, .k ∈ N, by uk (x) := Ex [Φk ϕ(zk )].

.

(3.6)

Theorem 3.1 The random evolution .uk (x) satisfies the following Markov renewal type equation: uk (x) = F x (k)D (x)ϕ(x) + k

.

k  

q(x, dy, s)D s (y)uk−s (y).

s=0 E

Proof We have Ex [Φk ϕ(zk )1{τ1 >k} ] = F x (k)D k (x)ϕ(x),

.

(3.7)

50

3 Discrete-Time Semi-Markov Random Evolutions

and Ex [Φk ϕ(zk )1{τ1 ≤k} ] =

k  

q(x, dy, s)Ex [Φk ϕ(zk ) | x1 = y, τ1 = s]

.

=

s=0 E k  

q(x, dy, s)D s (y)Ey [Φk−s ϕ(zk−s )],

s=0 E



and the result follows.

.

3.4 Discrete-Time Stochastic Systems In this section, we give some applications of the above results: to additive functionals that have many applications, e.g., in storage, reliability, and risk theories (see, e.g., [12, 90, 91, 103]); to geometric Markov renewal processes that also have many applications, including finance (see, e.g., [159–161, 163]), and dynamical systems or difference equations that are the discrete-time version of the dynamical differential systems with a huge number of applications (see, e.g., [19, 20]). We will consider here a semi-Markov chain .(zk , k ∈ N), with state space the measurable space .(E, E ) and semi-Markov kernel .q(x, B, k), .x ∈ E, .B ∈ E . Let also .νk := inf{n ≥ 0 : τn ≤ k}, the number of jumps in .[1, k] ⊂ N.

3.4.1 Additive Functionals Additive stochastic functionals of stochastic processes are important in theory and practice, as mean values in probability and statistics, as reward of stochastic systems, etc. There is a huge literature in the case of Markov chains additive functionals. In discrete-time semi-Markov case, there is a few of works only. Let us define the continuous bounded function .a : E → R and define the following additive functional (AF): yk = u +

k 

.

a(zs ),

k ≥ 0,

y0 = u.

s=1

If we define the operators .D(x), .x ∈ E, on .C0 (R) in the following way: D(x)ϕ(u) := ϕ(u + a(x)),

.

(3.8)

3.4 Discrete-Time Stochastic Systems

51

then the discrete-time semi-Markov random evolution .Φk ϕ can be written as follows: Φk ϕ(u) = ϕ(yk ).

.

Another additive functional can be defined, based on the embedded Markov chain, .(xn ), as follows: αk = u +

νk 

.

a(xs ),

k ≥ 0,

α0 = u.

(3.9)

s=1

This in fact is a Markov chain functional stopped in a random time .νk . The additive functional (3.8) can be represented in an alternative manner as follows: yk = u +

νk 

.

a(xn−1 )θn + a(xνk )(k − τνk ),

k ∈ N.

(3.10)

n=1

Let us define the random evolution .Φk := E[eisyk | zv , v ≤ k]. Then we have Φk =

νk

.

eisa(xn )θn+1 eisa(zk )γk ,

n=0

where .γk := k − τνk , we put for simplicity .u = 0. Define now the mean random evolution .u(x, k) := Ex [eisyk ] = E[eisyk | z0 = x], and .k = 0 is a jump time, i.e., .γ0 = 0. Then we can see that this random evolution satisfies the following Markov renewal equation: u(x, k) = F x (k)e

.

iska(x)

+

k  

q(x, dy, v)eisva(x) u(y, k − v).

(3.11)

v=0 E

Compare this Markov renewal equation with Markov renewal equation (3.7). We can also consider the following generalizations: αk = u +

νk 

.

a(xn , θn+1 ),

k ∈ N.

(3.12)

k ∈ N,

(3.13)

n=0

The following functional βk =

νk 

.

a(xn−1 , xn , θn+1 ),

n=1

in continuous-time, is considered by Pyke and Schaufele in [134].

52

3 Discrete-Time Semi-Markov Random Evolutions

3.4.2 Geometric Markov Renewal Chains Geometric Markov renewal processes are widely used in financial mathematics last years, (see, e.g., [159, 160]), and references therein. By taking the logarithm, these processes can be represented as additive functionals, but a direct treatment, at least for asymptotic analysis, gives more interesting results. These processes will be called in the case of discrete-time, Geometric Markov renewal chains (GMRC). Chapter 8 is dedicated to the optimal stopping of GMRC and their application in pricing. Let us consider a continuous bounded function .a : E → R, such that .a(x) > −1. The geometric Markov renewal chain (GMRC) is defined in the following way: Sk := S0

k

.

(1 + a(zj )),

k ∈ N∗ ,

S0 = s.

(3.14)

j =1

 We put that . 0k=1 = 1. If we define the operator .D(z) on .C0 (R) in the following way: D(z)ϕ(s) := ϕ(s(1 + a(z))),

.

then the discrete-time semi-Markov random evolution .Φk ϕ has the following presentation: Φk ϕ(s) = ϕ(Sk ).

.

We can also consider a GMRC defined with respect to the embedded Markov chain (.xn ) as Vk = V0

νk

.

(1 + a(xs )),

k ≥ 1, V0 = s.

s=1

This kind of GMRC is studied in particular in [160].

3.4.3 Dynamical Systems Dynamical systems of course is an important part in applied sciences and engineering modelling. The stochastic dynamical systems add an important factor in capacity of modelling since it allows one not only to get the mean value (deterministic case) but also to take into account uncertainties arising around their mean values. These systems can be extended also to stochastic differential equations by adding a

3.5 Discrete-Time Stochastic Systems in Series Scheme

53

white noise. The discrete-time form, knowing also as difference equation, is of great important, and their asymptotic behaviour will be the object in the next chapters. We consider here discrete-time dynamical systems (DS) and their asymptotic behaviour in series scheme: average and diffusion approximation, see [101]. Let us consider the difference equation uk+1 = uk + C(uk ; zk+1 ),

.

k ≥ 0,

and

u0 = u,

(3.15)

switched by the SMC .(zk ). The operators .D(x), x ∈ E, are defined now by D(x)ϕ(u) = ϕ(u + C(u, x)),

.

and the RE can be written as previously. The above difference equation can also be written as a recursive additive functional as follows: uk = u0 +

k−1 

.

C(us−1 ; zs ).

s=1

3.5 Discrete-Time Stochastic Systems in Series Scheme The limit behaviour of stochastic systems that we will study in the next chapters is presented in series scheme. That is, the stochastic systems presented in the previous section and of course the random evolutions will be indexed by a parameter .ε > 0. For each fixed .ε > 0, we have a stochastic system, and the limit results are obtained by letting .ε go to 0. For example, the additive functional in series scheme can be written as follows: ytε := ε

[t/ε] 

.

a(zs ),

t ∈ R+ ,

ε > 0,

s=1

where .[x] is the integer part of the positive real number x. Another usual parametrization is with parameter .n ∈ N. The relation between .ε and n can be .ε = 1/n, .n ≥ 1, and .ε goes to zero as n goes to infinity. So, for the above functional, we can write [nt]

n .yt

1 a(zs ), := n

n ≥ 1.

s=1

We prefer parameter .ε since it is more adapted to singular perturbation problems.

54

3 Discrete-Time Semi-Markov Random Evolutions

These functionals are studied, in the next chapters, in a functional way in the Skorokhod space .DE [0, ∞), or in .CE [0, ∞), where .E = Rd , .d ≥ 1, or .E = B. And where the weak convergence is denoted by . ⇒, that is, for the processes .ξtε , we write .ξtε ⇒ ξt0 as .ε → 0, if the corresponding law converges, that is, P ◦ (ξtε )−1 ⇒ P ◦ (ξt0 )−1 ,

.

as

ε → 0.

3.6 Concluding Remarks In this chapter, we presented the basic notion of random evolutions in discretetime in Banach spaces, discrete-time stochastic systems, and also considered their behaviour in series scheme with a functional setting in Skorokhod spaces. The basic operators that we used to describe stochastic systems are given. These operators are particular cases and satisfy the properties of general operators introduced to construct the random evolutions. This chapter was a preparation for the next chapter concerning the weak convergence of random evolutions. Many other stochastic systems can be considered almost in the same way. For further reading on this chapter the following references are of interest [57, 60, 101, 129, 144].

Chapter 4

Weak Convergence of DTSMRE in Series Scheme

4.1 Introduction In this chapter, we consider the general case of discrete-time semi-Markov random evolution, and we present their limit theory in series scheme. The limit results obtained here are abstract limit theorems of operators in Banach spaces, defining the random evolutions (see below Theorems 4.1, 4.2, and 4.3) that, in special cases, represent diffusions or functions as solutions of Cauchy problems deterministic or stochastic switched ones (see Sect. 4.4). In particular, we get weak convergence theorems in Skorokhod space .D[0, ∞) for càdlàg stochastic processes, see, e.g., [73]. The limit theorems include averaging and diffusion approximation. The last ones are of two kinds. The first one concerns equilibrium about a point, where a balance condition is needed, and the second one equilibrium about a function (or stochastic process) obtained by the averaging limit. This last case is called diffusion approximation with equilibrium or normal deviations. We also present some results on convergence rates of the above limit theorems. Finally, we give some applications of the above results, in particular to additive functionals, geometric Markov renewal chains and dynamical systems or difference equations, as well properties for empirical estimation of stationary probability of semi-Markov chains and to U -statistics. The first three cases have been already presented in Chap. 3 and considered also in [101, 104]. Applications of these stochastic systems are encountered in many problems in applied sciences and engineering, e.g., in storage, reliability and risk theories, statistics (see, e.g., [12, 90, 91, 103]), in finance (see, e.g., [159–161, 163]), in mechanics and statistics as dynamical systems (see, e.g., [27, 88]). The method of proofs is based on convergence of transition operators of the extended semi-Markov chain via a martingale characterization and solution of a singular perturbation problem. The tightness of the family of processes is proved via Sobolev’s embedding theorems [2, 151]. It is worth noticing that as in the Markov © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0_4

55

56

4 Weak Convergence of DTSMRE in Series Scheme

case, the results presented here cannot be deduced directly from the continuous-time case. Results presented in this chapter could be useful also for many applications where semi-Markov chain is the underlying model. For example, the additive functionals in performance analysis, and the geometric Markov renewal processes in finance and insurance, in statistics, in survival analysis, etc. This chapter is organized as follows. Averaging, diffusion approximation, and normal deviations of DTSMRE, as well as rates of convergence, are considered in Sect. 4.2. Section 4.3 presents the proof of theorems. Section 4.4 presents the applications.

4.2 Weak Convergence Results In this section, we present average and diffusion approximation results for the discrete-time semi-Markov random evolution, as well as diffusion approximation with equilibrium or normal deviations.

4.2.1 Averaging Let us set .k := [t/ε], and consider the continuous-time process .Mtε ε .Mt

:= M[t/ε] =

ε Φ[t/ε]

−I −

[t/ε]−1 

[P  D ε (·) − I ]Φε .

=0

We will prove here asymptotic results for this process as .ε → 0. The following assumptions are needed for averaging: A1: The MC .(zk , uk , k ∈ N) is uniformly ergodic with ergodic distribution π  (B × {k}), B ∈ E , k ∈ N.

.

A2: The moments .m2 (x), x ∈ E, are uniformly integrable, that is, .

lim sup

M→∞ x∈E



k 2 fx (k) = 0.

(4.1)

k≥M

A3: Let us assume that the perturbed operator .D ε (x) has the following representation in .B: D ε (x) = I + εD1 (x) + εD0ε (x),

.

4.2 Weak Convergence Results

57

where operators .D1 (x) on .B are closed and .B0 := ∩x∈E D(D  1 (x)) isdense in B, .B0 = B. Operators .D0ε (x) are negligible, that is, .limε→0 D0ε (x)ϕ  = 0 for .ϕ ∈ B0 .  A4: We have: . E π(dx) D1 (x)ϕ2 < ∞. A5: There exist Hilbert spaces .H and .H∗ such that compactly embedded in Banach spaces .B and .B∗ , respectively, where .B ∗ is a dual space to .B. A6: Operators .D ε (z) and .(D ε )∗ (z) are contractive on Hilbert spaces .H and .H∗ , respectively. We note that if .B = C0 (R), then .H = W l,2 (R) is a Sobolev space, and l,2 (R) ⊂ C (R), and this embedding is compact (see [151]). For the spaces .W 0 l,2 (R), the situation is the same. .B = L2 (R) and .H = W .

Theorem 4.1 Under Assumptions A1–A6, the following weak convergence takes place ε  Φ[t/ε]

⇒ Φ(t),

.

ε ↓ 0,

 is determined by the following equation: where the limit random evolution .Φ(t)  Φ(t)ϕ −ϕ−



t

.

  LΦ(s)ϕds = 0,

0 ≤ t ≤ T,

ϕ ∈ B0 ,

(4.2)

0

or, equivalently .

d   Φ(t)ϕ = LΦ(t)ϕ, dt

where the limit contracted operator is then given by  1 = L=D

 π(dx)D1 (x).

.

(4.3)

E

4.2.2 Diffusion Approximation For the diffusion approximation, we will consider a different time-scaling and some additional assumptions. Here we replace assumptions A2–A4 by the more implied assumptions below D1, D3–D4, and the new one D2 that express an equilibrium point that is needed for the diffusion approximation. D1: Let us assume that the perturbed operators .D ε (x) have the following representation in B: D ε (x) = I + εD1 (x) + ε2 D2 (x) + ε2 D0ε (x),

.

58

4 Weak Convergence of DTSMRE in Series Scheme

where operators .D2 (x) on B are closed and .B0 := ∩x∈E D(D2 (x)) is ε dense in .B 0 = B; operators .D (x) are a negligible operator, that is 0   B, ε   .limε↓0 D (x)ϕ = 0. 0 D2: The following balance condition holds Π D1 (x)Π = 0.

.

(4.4)

D3: The moments .m3 (x), x ∈ E, are uniformly integrable, that is, .

lim sup

M→∞ x∈E



k 3 fx (k) = 0.

(4.5)

k≥M

 D4: We have . E π(dx) D1 (x)ϕ4 < ∞. Theorem 4.2 Under Assumptions A1, A5–A6 (see Sect. 4.2.1), and D1–D4, the following weak convergence takes place ε Φ[t/ε 2 ] ⇒ Φ0 (t),

.

ε ↓ 0,

where the limit random evolution .Φ0 (t) is a diffusion random evolution determined by the following generator: L = Π D2 (x)Π + Π D1 (x)R0 D1 (x)Π − Π D12 (x)Π.

.

4.2.3 Normal Deviations We note that averaged semi-Markov random evolutions can be considered as the first approximation to the initial evolutions. The diffusion approximation of the semi-Markov random evolutions determines the second approximation of the initial evolution. Here we consider algorithms of construction of the first and second approximations in the case when the balance condition in the diffusion approximation scheme is not fulfilled. We introduce the deviated semi-Markov random evolution as the normalized difference between the initial and averaged evolutions. In the limit, we obtain the diffusion approximation with equilibrium of the initial evolution from the averaged one. We call it normal deviations. ε Let us consider the discrete-time semi-Markov random evolution .Φ[t/ε] , aver (see Sect. 3.1) and the deviated evolution aged evolution .Φ(t) ε  Wtε := ε−1/2 [Φ[t/ε] − Φ(t)].

.

(4.6)

4.2 Weak Convergence Results

59

Theorem 4.3 Under Assumptions A1, A3–A6 (see Sect. 4.2.1), and D3, the deviated semi-Markov random evolution .Wtε weakly converges, when .ε → 0, to the diffusion random evolution .Wt0 defined by the following generator: 1 )R0 (D1 (x) − D 1 )Π. L = Π (D1 (x) − D

.

(4.7)

4.2.4 Rates of Convergence in the Limit Theorems We present here the rates of convergence of DTSMRE in the averaging, diffusion approximation, and diffusion approximation with equilibrium schemes, and we give the rates of convergence for additive functionals and geometric Markov renewal chains, in the corresponding limits as corollaries (see [91, 156, 157]). Proposition 4.1 The rate of convergence of DTSMRE in the averaging has the following form: .

    ε  ϕ] − Φ(t)ϕ E[Φ[t/ε]  ≤ εA(T , ϕ, R0  , D1 ),

where .A(T , ϕ, R0  , D1 ) is a constant, and .0 ≤ t ≤ T . The proof of this proposition is given in Sect. 4.4. Proposition 4.2 The rate of convergence of DTSMRE in the diffusion approximation takes the following form: .

    ε E[Φ[t/ε 2 ] ϕ] − Φ0 (t)ϕ  ≤ εD(T , ϕ, R0  , D1  , D2 ),

where .D(T , ϕ, R0  , D1  , D2 ) is a constant, and .0 ≤ t ≤ T . Proposition 4.3 The rate of convergence of DTSMRE in diffusion approximation with equilibrium has the following form: .

  √   E[Wtε ϕ] − Wt0 ϕ  ≤ εN(T , ϕ, R0  , D1  , D12 ),

  where .N (T , ϕ, R0  , D1  , D12 ) is a constant and .0 ≤ t ≤ T . The proofs of the above Propositions 4.2 and 4.3 are similar to the proof of Proposition 4.1.

60

4 Weak Convergence of DTSMRE in Series Scheme

4.3 Proof of Theorems We prove here the weak convergence in limit theorems via martingale representation and singular perturbing problems into two steps. • Step 1: We prove the relative compactness of the family of processes by establishing the compact containment condition and then the relative compactness of ε ∗ ∗ .(Φt ), . ∈ B (where .B is a dual space of .B that separates points). • Step 2: We prove the convergence of the perturbed transition operator of the extended semi-Markov renewal chains by solving a singular perturbation problem.

4.3.1 Proof of Theorem 4.1 Let us first prove the relative compactness of DTSMRE in the average scheme. This proof is included into the four lemmas proved here. And in a final step, we prove the theorem. ε Let us consider the DTSMRE .Φ[t/ε] ϕ, as defined by relation (3.4), which is weakly compact in .DB [0, ∞) with limit points into .CB [0, ∞). The proof is based on the following lemmas. ε Lemma 4.1 Under conditions of Theorem 4.1, the limit points of .Φ[t/ε] ϕ, .ϕ ∈ B0 , as .ε → 0, belong to .CB [0, ∞).

Proof We note that from conditions A5–A6 it follows that discrete-time semiMarkov random evolution .Φk ϕ is a contractive operator in .H, and hence, .Φk ϕH is a supermartingale for any .ϕ ∈ H, where .·H is a norm in Hilbert space .H ε ([91, 151]). Obviously, the same properties satisfy the following family .Φ[t/ε] . Using    ε  Doob’s inequality for the supermartingale .Φ[t/ε]  , we can get that H

ε P(Φ[t/ε] ∈ KΔ ) ≥ 1 − Δ,

.

where .KΔ is a compact set in .B and .Δ is any small number. This means that ε sequence .Φ[t/ε] is tight in .B. Taking into account conditions A1–A6, we obtain that ε discrete-time semi-Markov random evolution .Φ[t/ε] is weakly compact in .DB [0, ∞) with limit points in .CB [0, ∞), .ϕ ∈ B0 . Let     ε ε ε ε .Jt := J (Φ[t/ε] ; [t/ε]) := sup Φ[t/ε]+k ϕ − Φ[t/ε] ϕ  , k≤[t/ε]

4.3 Proof of Theorems

61

and let .KΔ be a compact set from compact containment condition .Δ > 0. It is sufficient to show that .Jtε weakly converges to zero. This is equivalent to the convergence of .Jtε in probability as .ε → 0. From the very definition of .Jtε and A3, we obtain   Jtε 1KΔ ≤ ε sup sup (D1 (zk )ϕ + D0ε (zk )ϕ ),

.

k≤[t/ε] ϕ∈SΔ

where .1KΔ is the indicator of the set .KΔ , and .SΔ is the finite .δ-set for .KΔ . Then for δ < Δ, we have

.

Pπ (Jtε 1KΔ > Δ) ≤ Pπ ( sup Dk > (Δ − δ)/ε)

.

k≤[t/ε]

=

[t/ε] 

Pπ ({ sup Dk > (Δ − δ)/ε} ∩ Di ) k≤[t/ε]

i=1

   ≤ ε2 [t/ε] sup (P  )[t/ε] (D1 (x)ϕ2 + 2 D1 (x)ϕ D0ε (x)ϕ  ϕ∈SΔ

 2  + D0ε (x)ϕ  ) ,   where .Dk := supϕ∈SΔ (D1 (zk )ϕ + D0ε (zk )ϕ ), and Di := {ω : Dk contains the maximum for the first time on the variable Di }.

.

It is worth noticing that the operator .(P  )k is bounded when .k → ∞. So is the case for .(P  )[t/ε] when .ε → 0. . Taking both .ε and .δ go to 0, we obtain the proof of this lemma. Let us now consider the continuous-time martingale ε .Mt

:=

ε M[t/ε]

=

ε Φ[t/ε]

−I −

[t/ε]−1 

ε Eπ [Φk+1 − Φkε | Fk ].

k=0

Lemma 4.2 The process ε .Mt

:=

ε Φ[t/ε]

−I −

[t/ε]−1  =0

is an .F[t/ε] -martingale.

[P  D ε (·) − I ]Φε

(4.8)

62

4 Weak Convergence of DTSMRE in Series Scheme

Proof As long as Mkε := Φkε − I −

.

k−1  [P  D ε (·) − I ]Φε =0

ε ε is a martingale, .Mtε = M[t/ε] is an .F[t/ε] -martingale. Here we have .Eπ [Mk+1 | ε Fk ] = Mk , which can be easily checked. . [t/ε] ε ϕ − Φ ε ϕ | F ]) is relatively compact for Lemma 4.3 The family .( k=0 Eπ [Φk+1 k k ∗ all . ∈ B0 , dual of the space .B0 .

Proof Let Ntε :=

[t/ε] 

.

ε Eπ [(Φk+1 − Φkε )ϕ | Fk ].

k=0

Then Ntε =

[t/ε] 

.

[P  D ε (·) − I ]Φkε .

k=0

As long as .Φk+1 = D ε (zk+1 )Φkε , we obtain ε Eπ [Φk+1 ϕ | Fk ] = Eπ [D ε (zk+1 )Φkε ϕ | Fk ].

.

Then





[(t+η)/ε]



[(t+η)/ε]  





ε ε  ε ε





. ( Eπ [Φk+1 ϕ − Φk ϕ | Fk ]) = ( [P D (zk+1 ) − I ]Φk ϕ)



k=[t/ε]+1



k=[t/ε]+1 ≤ ε  ([(t + η)/ε] − [t/ε] − 1)   × P  (D1 (zk+1 ) + D0ε (zk+1 ))ϕ   η ≤ ε  P  (D1 (·) + D0ε (·))ϕ  ε    = η  P (D1 (·) + D ε (·))ϕ  0

→ 0,

η → 0,

  since .P  (D1 (·) + D0ε (·))ϕ  is bounded for any .ϕ ∈ B0 . [t/ε] ε ϕ − Φ ε ϕ | F ]) is relatively compact It means that the family .( k=0 Eπ [Φk+1 k k ∗ for any . ∈ B0 .

4.3 Proof of Theorems

63

ε ϕ) is relatively compact for any . ∈ B∗ , and any Lemma 4.4 The family .(M[t/ε] 0 .ϕ ∈ B0 . ε Proof It is worth noticing that the martingale .M[t/ε] can be represented in the form of the martingale differences

ε M[t/ε] =

[t/ε]−1 

.

ε ε Eπ [(Φk+1 ϕ − Eπ (Φk+1 ϕ | Fk )].

k=0

Then, using the equality ε Eπ [Φk+1 ϕ | Fk ] = Eπ [D ε (zk+1 )Φkε ϕ | Fk ],

.

we obtain ε ε M[(t+η)/ε] ϕ − M[t/ε] ϕ=

[(t+η)/ε] 

.

[D ε (zk+1 )Φkε ϕ − Eπ [D ε (zk+1 )Φkε ϕ | Fk ]}

k=[t/ε]+1

=

[(t+η)/ε] 

[D ε (zk+1 Φkε ϕ − P  D ε (zk+1 )Φkε ϕ]

k=[t/ε]+1

=

[(t+η)/ε] 

[D ε (zk+1 ) − P  D ε (zk+1 )]Φkε ϕ,

k=[t/ε]+1

for any .η > 0. Now, from the above, we obtain





ε ε Eπ (M[(t+η)/ε] ϕ − M[t/ε] ϕ) ≤ ([t + η)/ε] − [t/ε])εEπ (D1 (zk+1 )ϕ       + D0ε (zk+1 )ϕ  + P  D1 (·)ϕ  + P  D0ε (·)ϕ )     ≤ 2η(P  D1 (·)ϕ  + P  D0ε (·)ϕ )

.

→ 0,

η → 0,

and the proof is completed.



.

Now the proof of Theorem 4.1 is achieved as follows. From Lemmas 4.2, 4.3, and 4.4 and the representation (4.8), it follows that the ε ϕ) is relatively compact for any . ∈ B∗0 , and any .ϕ ∈ B0 . family .(Φ[t/ε] ε Let .L (x), .x ∈ E, be a family of perturbed operators defined on .B by Lε (x) := ε−1 Q + P  D1 (x) + P  D0ε (x).

.

64

4 Weak Convergence of DTSMRE in Series Scheme

Then the process [t/ε]−1 

ε Mtε = Φ[t/ε] −I −ε

.

Lε Φε

=0

is an .Ftε -martingale. The following singular perturbation problem, for the non-negligible part of compensating operator, .Lε , denoted by .Lε0 (x) := ε−1 Q + P  D1 (x), Lε0 ϕ ε = Lϕ + εθ ε ,

.

on the test functions .ϕ ε (u, x) = ϕ(u) + εϕ1 (u, x), has the solution (see Proposition 1.1, Chap. 1, and also in [90] Proposition 5.1): 1 ϕ, ϕ1 = R0 D

ϕ ∈ NQ ,

.

with 1 , 1 (x) = P  D1 (x) − D D  1 = π(dx)D1 (x), D

.

E

1 (x)ϕ. θ (x) = P  D1 (x)R0 D ε

The limit operator is then given by LΠ = Π D1 (·)Π,

.

from which we get the contracted limit operator  1 . L=D

(4.9)

.

We note that martingale .Mtε has the following asymptotic representation: ε Mtε = Φ[t/ε] −I −ε

[t/ε]−1 

.

 LΦε + Oϕ (ε),

=0

  where .Oϕ (ε) → 0, as .ε → 0. The families

(M[t/ε] )

.

and

(

[t/ε]−1  =0

[P  D ε (·) − I ]Φε )

(4.10)

4.3 Proof of Theorems

65

are weakly compact for all . ∈ B∗0 in a dense subset .B∗0 ⊂ B. This means that family [t/ε]−1 ε  LΦε ϕ converges, .(Φ [t/ε] ) is also weakly compact. In this way, the sum .ε =0 t  as .ε → 0, to the integral . 0  LΦ(s)ϕds. The quadratic variation of the martingale ε ε .(Mt ϕ) tends to zero when .ε → 0; hence, .Mt ϕ → 0, when .ε → 0, for any .ϕ ∈ B0 ∗ and for any . ∈ B0 . Passing to the limit in (4.10), when .ε → 0, we obtain the desired result. In the average scheme, the limit .Mt0 , for the martingale .Mtε , has a quadratic variation equal to 0. In fact, the quadratic variation is ε (M[t/ε] ) =

[t/ε] 

.

ε Eπ [2 (Mk+1 ϕ ε − Mkε ϕ ε ) | Fk ],

(4.11)

k=0

where .ϕ ε (x) = ϕ(x) + εϕ1 (x). Hence, ε ε ε (Mk+1 ϕ ε − Mkε ϕ ε ) = ((Mk+1 − Mkε )ϕ) + ε((Mk+1 − Mkε )ϕ1 ),

.

and ε ε ε Mk+1 − Mkε = Φk+1 − Φkε − Eπ [Φk+1 − Φkε | Fk ].

.

Hence, ε (Mk+1 ϕ ε − Mkε ϕ ε ) = ((D ε (zk+1 ) − I )Φkε ϕ) − Eπ [(D ε (zk+1 ) − I )ϕ | Fk ]

.

+ε((D ε (zk+1 ) − I )ϕ1 ) − Eπ [(D ε (zk+1 ) − I )ϕ1 | Fk ] = ε((D1 (zk+1 ) + D0ε (zk+1 ))ϕ)

(4.12)

−εEπ [(D1 (zk+1 ) + D0ε (zk+1 ))ϕ | Fk ] +ε2 ((D1 (zk+1 ) + D0ε (zk+1 ))ϕ1 ) −ε2 Eπ [(D1 (zk+1 ) + D0ε (zk+1 ))ϕ1 | Fk ]. From (4.11) and (4.12) and from boundedness of all operators in (4.12) with ε ) converges, when .ε → 0, to 0. respect to .Eπ , it follows that .(M[t/ε] In this case, the limit martingale .Mt0 equals to 0. So, the limit equation for .Mtε has the form (4.1). As long as the solution of the martingale problem for operator L is unique, then it follows that the solution of the equation (4.1) is unique as well. . 1 , see (4.9)). Finally, L is a first-order operator (.D It is worth noticing that operator .   Lt], and the latter the operator .L generates a semigroup, and then .Φ(t)ϕ = exp[ representation is unique. .

66

4 Weak Convergence of DTSMRE in Series Scheme

4.3.2 Proof of Theorem 4.2 ε We can prove the relative compactness of the family .Φ[t/ε 2 ] exactly on the same way, and following the same steps as in the proof of Theorem 4.1. But in the case of diffusion approximation, the limit continuous martingale .M0 (t) for the martingale ε .Mt has quadratic variation that is not zero. That is



t

M0 (t)ϕ = Φ0 (t)ϕ − ϕ −

.

 LΦ0 (s)ds,

0 ε  = 0, for . ∈ B∗ . and so .(M[t/ε] 0 L defined in Theorem 4.2 is a second-order kind Furthermore, the operator . 2 and .Π D1 R0 P  D1 Π (compare with the operator since it contains the operator .D L in (4.9)). first -order operator . Let .Lε (x), .x ∈ E, be a family of perturbed operators defined on .B as follows:

Lε (x) := ε−2 Q + ε−1 P  D1 (x) + P  D2 (x) + P  D0ε (x).

.

Then the process ε .Mt

=

ε Φ[t/ε ε]

−I −ε

2

2 ]−1 [t/ε 

Lε Φkε

k=0

is an .Ftε -martingale with mean value 0. For the non-negligible part of compensating operator, .Lε , denoted by Lε0 (x) := ε−2 Q + ε−1 P  D1 (x) + P  D2 (x),

.

consider the following singular perturbation problem: Lε0 ϕ ε = Lϕ + εθ ε (x),

.

(4.13)

where .ϕ ε (u, x) = ϕ(u) + εϕ1 (u, x) + ε2 ϕ2 (u, x). The solution of this problem is realized by the vectors (see Proposition 1.2 in Chap. 1, and also in [90], Proposition 5.2) ϕ1 = R0 P  D1 (x)ϕ,

.

 ϕ2 = R0 Aϕ,

 := A(x) − A,  and the negligible term .θ ε (x) is with .A(x) θ ε (x) = [P  D1 (x) + εP  D2 (x)]ϕ2 + P  D2 (x)ϕ1 .

.

Of course, .ϕ ∈ N(Q ).

4.3 Proof of Theorems

67

Now the limit operator .L is given by L = P  D2 (·) + P  D1 (·)R0 P  D1 (·),

.

from which the contracted operator on the null space .N(Q ) is  2 Π + Π D1 (x)R0 P  D1 (x)Π. L=D

.

Using the balance condition (4.4), we get the limit operator. We note that from conditions A5–A6 and D1-D2 it follows that discrete-time ε semi-Markov random evolution .Φ[t/ε 2 ] ϕ is a contractive operator in .H and, hence,    ε  .Φ ϕ  is a supermartingale for any .ϕ ∈ H, where .·H is a norm in Hilbert [t/ε2 ] H    ε  space .H ([91, 151]). Using Doob’s inequality for the supermartingale .Φ[t/ε 2] , H we can get that ε 1 P(Φ[t/ε 2 ] ∈ KΔ , 0 ≤ t ≤ T ) ≥ 1 − Δ,

.

1 is a compact set in B and .Δ is any small number. where .KΔ It means that under conditions A5–A6 and D1–D2, the sequence .Mtε is tight and is weakly compact in .DB [0, ∞) with limit points in .CB [0, ∞). Also, that under conditions A5–A6 and D1–D2, the martingale .Mtε has the following asymptotic presentation:

ε .Mt ϕ

=

ε Φ[t/ε 2]ϕ

−ϕ−ε

2

2 ]−1 [t/ε 

 LΦkε ϕ + Oϕ (ε),

(4.14)

k=0

  [t/εε ]−1  where .Oϕ (ε) → 0, as .ε → 0. The families .(Mtε φ) and .(ε2 k=0 LΦkε ϕ) ε ∗ are weakly compact for all . ∈ B and .ϕ ∈ B0 . This means that .Φ[t/ε2 ] is also weakly compact and has a limit. If we set .Φ0 (t) for this limit, then the sum t [t/εε ]−1 2  LΦkε ϕ converges to the integral . 0  LΦ0 (s)ϕds. Let also .M0 (t) be a .ε k=0 limit martingale for .Mtε when .ε → 0. Then from the previous steps and (4.14), we see that  t  LΦ0 (s)ϕds. .M0 (t)ϕ = Φ0 (t)ϕ − ϕ − (4.15) 0

As long as the martingale .Mtε has mean value zero, the martingale .M0 (t) has also mean value zero. Taking the mean value from both parts of (4.15), we get 

t

0 = EΦ0 (t)ϕ − ϕ −

.

0

 LEΦ0 (t)ϕds.

(4.16)

68

4 Weak Convergence of DTSMRE in Series Scheme

Solving this, we obtain EΦ0 (t)ϕ = exp[ Lt]ϕ.

(4.17)

.

L generates the semigroup .U (t) := EΦ0 (t)ϕ = This means that the operator . exp[ Lt]. The uniqueness of the limit evolution .Φ0 (t) in diffusion approximation scheme follows from the uniqueness of solution to the martingale problem for .Φ0 (t). L is unique, then As long as the solution of the martingale problem for the operator . it follows that the solution of (4.17) is unique as well[45, 153]. .

4.3.3 Proof of Theorem 4.3 We note that .Wtε in (4.6) has the following presentation: ε .Wt

=ε

−1/2

{

[t/ε] 



[D

ε

(zk−1 ) − I ]Φkε

k=1

−

t

1 Φ(s)ds}. D

(4.18)

0

1 ) = 0, then the diffusion approximation algorithm (see Sect. 4.2) Since .Π (D1 − D may be applied to the right-hand side of (4.18) with the operators: .D2 = 0 and 1 ) instead of .D1 (z). We mention that the sequence .Wtε is weakly .(D1 (z) − D . compact, and the result is proved (see Sects. 4.4.1–4.4.2).

4.3.4 Proof of Proposition 4.1 The problem is to estimate .

    ε  ϕ ε ] − Φ(t)ϕ Eπ [Φ[t/ε] ,

for any .ϕ ∈ B0 , where .ϕ ε (x) = ϕ(x) + εϕ1 (x). We note that (P  − I )ϕ1 (x) = −(Dˆ 1 − P  D1 (x))ϕ.

.

(4.19)

As long as .Π (Dˆ 1 − P  D1 (x))ϕ = 0, .ϕ ∈ B0 , the solution to (4.19) is in domain R(P  − I ), .ϕ1 (x) = R0 D˜ 1 ϕ. Then    .Eπ ϕ1 (x) ≤ 2 R0  π(dz) P  D1 (z)ϕ  =: 2C1 (ϕ, R0 ), (4.20)

.

E

4.3 Proof of Theorems

69

where .R0 is the potential operator of .Q := P  − I . From here, we obtain    ε  Eπ (Φ[t/ε] − I )ϕ1  ≤ 4C1 (ϕ, R0 ),

.

(4.21)

since the .Φkε are contractive operators. We also note that      t [t/ε]    ε  ˆ ˆ ¯ LΦk ϕ − LΦ(s)ϕds] . Eπ [ε  ≤ εC2 (t, ϕ),  0   k=0 where 

  π(dz) P  D1 (z)ϕ  ,

C2 (t, ϕ) := 4T

.

t ∈ [0, T ].

E

This follows from a standard argument about the convergence of Riemann sums in Bochner integral (see Lemma 4.14 of [91, p. 161]). We have         ε ε ε   . Eπ [Φ[t/ε] ϕ ] − Φ(t)ϕ ϕ − Φ(t)ϕ  ≤ Eπ [Φ[t/ε]  +εC1 (ϕ, R0 ),

(4.22)

 satisfies the where we have used representation .ϕ ε = ϕ + εϕ1 . Note that .Φ(t) equation  Φ(t)ϕ −ϕ−



t

.

ˆ Φ(s)ϕds  L = 0.

0

Let us introduce the following martingale: ε ε .M[t/ε]+1 ϕ

:=

−ϕ −

ε Φ[t/ε] ϕε

ε

[t/ε] 

ε Eπ [Φk+1 ϕ ε − Φkε ϕ ε | Fk ].

(4.23)

k=0

This is a zero-mean value martingale, that is, ε Eπ M[t/ε] ϕ ε = 0,

.

which follows directly from (4.23).

(4.24)

70

4 Weak Convergence of DTSMRE in Series Scheme

Again, from (4.23), we get the following asymptotic representation: ε ε M[t/ε] ϕ ε = Φ[t/ε] ϕ − ϕ + ε[Φ[t/ε] − I ]ϕ1 − ε

[t/ε] 

.

ˆ εϕ LΦ k

k=0

−ε2

[t/ε] 

[P  D1 (·)Φkε ϕ1 + oϕ (1)],

(4.25)

k=0

where .oϕ (1) → 0, as .ε → 0, for any .ϕ ∈ B0 . Now, from (4.2) and expressions (4.24)–(4.25), we obtain ε ¯ Eπ [Φ[t/ε] ϕ − Φ(t)ϕ] = εEπ [Φ[t/ε]ε − I ]ϕ1 + Eπ [ε

[t/ε] 

.

ˆ εϕ LΦ k

k=0



t

−

ˆ Φ(s)ϕds] ¯ L + ε2 Eπ [

[t/ε]−1 

0

Rk (ϕ1 )], (4.26)

k=0

where .Rk (ϕ1 ) := P  D1 (·)Φkε ϕ1 + oϕ (1). Let us estimate .Rk (ϕ1 ) in (4.26). We have .

    Rk (ϕ1 ) ≤ sup (P  D1 (z)g  + og (1)) =: C3 (z, g, KΔ ),

(4.27)

g∈KΔ

where .KΔ is a compact set, .Δ > 0, because .Φkε ϕ1 satisfies compactness condition for any .ε > 0 and any k. In this way, we get from (4.26) that      [t/ε]−1    ≤T  . Eπ [ R (ϕ )] π(dz)C3 (z, g, KΔ ), k 1   E   k=0

t ∈ [0, T ].

(4.28)

Finally, from inequalities (4.20), (4.21), and (4.22) and from (4.27)–(4.28), we obtain the desired rate of convergence of the DTSMRE in the averaging scheme, that is,     ε ε  . Eπ [Φ[t/ε] ϕ ] − Φ(t)ϕ  ≤ εA(T , ϕ, R0  , D1 ), where the constant  A(T , ϕ, R0  , D1 ) := 5C1 (ϕ, R0 ) + C2 (T , ϕ) + T

π(dz)C3 (z, g, KΔ ),

.

E

and .C3 (z, g, KΔ ) is defined in (4.27). This completes the proof of Proposition 4.1. .

4.4 Applications of the Limit Theorems to Stochastic Systems

71

In a similar way, we can obtain the rate of convergence results in the diffusion approximation (see Propositions 4.2–4.3).

4.4 Applications of the Limit Theorems to Stochastic Systems In this section, we give applications of the Theorems 4.1, 4.2, and 4.3. Namely, we present asymptotic results in series scheme of additive functionals, of geometric Markov renewal processes, of dynamical systems and some additional results for an empirical estimator of stationary probability of SMCs and U -statistics. Consider the standard Brownian motion or Wiener process, .(wt , t ∈ R+ ), with .w0 = 0 and .Cov (wt , ws ) = t ∧ s, for .t, s ≥ 0. In this section, we consider a SMC .(zk ) with state space .(E, E ), semi-Markov kernel .q(x, B, k), and embedded Markov chain .(xn ). Moreover, we denote by .π the stationary probability of .(zk ) and by .ρ the stationary probability of the EMC .(xn ). The process .(νk ) counts the number of jumps of .(zk ) up to the time .k ∈ N. The same notation is used for series scheme, where the same symbols include an upper index .ε. See Chaps. 2 and 3.

4.4.1 Additive Functionals Let us define the following additive functional: yk = u +

k 

.

a(zs ),

k ≥ 0,

y0 = u.

s=1

Define the operator .D(z) on .B := C0 (R) by D(z)ϕ(y) := ϕ(y + a(z)),

.

and then the discrete-time semi-Markov random evolution .Φk ϕ is Φk ϕ(y) = ϕ(yk ).

.

Averaging Now, define the continuous-time process ytε := u + ε

[t/ε] 

.

s=1

a(zs ),

t ≥ 0,

ε > 0,

y0ε = u,

72

4 Weak Convergence of DTSMRE in Series Scheme

 and assume that . E π(dz) |a(z)| < ∞, and then, from Theorem 4.1, we obtain the following result. Proposition 4.4 The process .ytε converges weakly, as .ε → 0, to the . y (t), given by  y (t) = u +  a t,

.

where  .a =

 E

 y (0) = u,

π(dz)a(z). This is a deterministic function.

Diffusion Approximation Consider the continuous -time process .ξtε as follows: := u + ε

ε .ξt

2 [t/ε ]

ξ0ε = u,

a(zs ),

s=1

and then, from Theorem 4.2, we get the following result.   Proposition 4.5 Under the balance condition . E π(dz)a(z) = 0 and . E π(dz)|a (z)|2 < ∞, the process .ξtε converges weakly, as .ε → 0, to the process .ξ0 (t), given by ξ0 (t) = y + bwt ,

.

 where .b2 = 2aˆ 0 − aˆ 2 , aˆ 0 = E π(dz)a(z)R0 a(z), .wt is a standard Wiener process.

aˆ 2 =

 E

π(dz)a 2 (z), and

Normal Deviations For the normal deviations, or, diffusion approximation with equilibrium, let us consider the following normalized additive functional:  ytε := ε−1/2 [ytε −  a t].

.

Then, from Theorem 4.3, we obtain the result. Proposition 4.6 The process .y˜tε converges weakly, as .ε → 0, to the process .σ wt , where  2 .σ = π(dz)(a(z) − a)R ˆ 0 (a(z) − a), ˆ t ≥0 E

and .wt is a standard Wiener process. In this way, the AF .ytε may be presented in the following approximated form: ytε ≈  at +

.

for small .ε values.

√ εσ wt ,

t ≥ 0,

4.4 Applications of the Limit Theorems to Stochastic Systems

73

4.4.2 Geometric Markov Renewal Processes The geometric Markov renewal process (GMRP) was studied and applied in financial problems, see [159, 160, 163]. Now, in discrete time, the geometric Markov renewal chain (GMRC) is defined as follows: k

Sk := S0

.

(1 + a(zl )),

k ∈ N∗ ,

S0 = s.

l=1

Define the operator .D(z), on .B := C0 (R), by D(z)ϕ(s) := ϕ(s(1 + a(z))).

.

Then the discrete-time semi-Markov random evolution .Φk ϕ has the following presentation: Φk ϕ(s) = ϕ(Sk ).

.

It is worth noticing here that we are using .St instead of .ln(1 + St ) in order to be consistent with the discrete models for stock prices in mathematical finance proposed by Cox–Ross–Rubinstein [32] and Aase [1]. It is worth noticing that in finance the expression .ln(St /S0 ) represents the log-return of the underlying asset (stock, asset, etc.) .St . Averaging Now, define the sequence of processes Stε := S0ε

[t/ε]

(1 + εa(zk )),

.

t ∈ R+ ,

S0ε = s,

ε > 0.

k=1

Then, from Theorem 4.1, we obtain the result. Proposition 4.7 Under the averaging conditions, the process .Stε converges weakly, as .ε → 0, to the process .Sˆt  St = sea t ,

.

where  .a :=

 E

π(dz)a(z) < ∞, and this is a deterministic function.

Diffusion Approximation If we define the following sequence of processes: S (t) :=

.

ε

S0ε

2 [t/ε ]

(1 + εa(zk )),

t ∈ R+ ,

k=1

Then, from Theorem 4.2, we obtain the result.

S0ε = s,

ε > 0.

74

4 Weak Convergence of DTSMRE in Series Scheme

  Proposition 4.8 Under the balance condition . E ρ(dx) E P (x, dy)a(y) = 0, the process .S ε (t), in the diffusion approximation scheme, converges weakly, as .ε → 0, to the process .S0 (t) S0 (t) = se−t aˆ 2 /2 eσa w(t) ,

.

where  aˆ 2 :=

π(dz)a 2 (z),

.

E

 2 .σa

:=

π(dz)[a 2 (z)/2 + a(z)R0 a(z)], E

and .wt is a standard Wiener process. It means that .S0 (t) satisfies the following stochastic differential equation: dS0 (t) =

.

1 2 (σ − aˆ 2 )S0 (t)dt + σa S0 (t)dwt . 2 a

Normal Deviations Let us consider the following normalization of the GMRC: ˆ S˜tε := ε−1/2 [ln(Stε /S0ε ) − at],

.

t ∈ R+ ,

S0ε = s,

ε > 0.

Then, from Theorem 4.3, we obtain the result. Proposition 4.9 The process .S˜tε converges weakly, as .ε → 0, to the process .σ wt , where  2 .σ = π(dz)(a(z) − a)R ˆ 0 (a(z) − a), ˆ E

and .wt is a standard Wiener process. In this way, the GMRC .Stε may be presented in the following approximated form: √ εσ wt

ˆ Stε ≈ S0 eat+

.

.

It is worth noticing that convergence results for GMRC can be obtained directly via the additive functional by considering, instead of the function a, the function .ln(1 + a(x)).

4.4 Applications of the Limit Theorems to Stochastic Systems

75

4.4.3 Dynamical Systems We consider here discrete-time dynamical systems and their asymptotic behaviour in series scheme: average and diffusion approximation. Let us consider the difference equation ε yk+1 = ykε + εC(ykε ; zk+1 ),

.

k ≥ 0,

and

y0ε = u,

ε > 0,

(4.29)

switched by the SMC .(zk ). The perturbed operators .D ε (x), x ∈ E, are defined now by D ε (x)ϕ(u) = ϕ(u + εC(u, x)).

.

ε Averaging The averaging series scheme in (4.29) is the processes .(y[t/ε] ), that is, ε ε y[t/ε]+1 = y[t/ε] + εC(ykε ; z[t/ε]+1 ),

.

k ≥ 0,

and

y0ε = u,

ε > 0.

(4.30)

Then, from Theorem 4.1, we obtain the result. Proposition 4.10 Under averaging assumptions, the following weak convergence takes place ε y[t/ε] ⇒ y (t),

.

as

ε → 0,

where . y (t), t ≥ 0 is the solution of the following (deterministic) differential equation: .

 and .C(u) := function.

 E

d  y (t)),  y (t) = C( dt

and  y (0) = u,

(4.31)

π(dx)C(u, dx). The process . y (t), t ≥ 0, is a deterministic

Diffusion Approximation In the diffusion approximation, the process in series ε scheme is .y[t/ε 2]. Then, from Theorem 4.2, we obtain the result.  Proposition 4.11 Under the balance condition . E π(dx)C(u, x) ≡ 0, and  2 . E π(dx)C (u, x) < ∞, the following weak convergence takes place ε y[t/ε 2 ] ⇒ xt ,

.

as

ε → 0,

where .xt , t ≥ 0, is a diffusion process, with initial value .x0 = u, determined by the operator 1 Lϕ(u) = a(u)ϕ  (u) + b2 (u)ϕ  (u), 2

.

76

4 Weak Convergence of DTSMRE in Series Scheme

provided that .b2 (u) > 0, and drift and diffusion coefficients are defined as follows: .

b2 (u) := 2C 0 (u) − C 2 (u), a(u) := C 01 (u) − C 1 (u),

with:

 C 0 (u) := E π(dx)C0 (u, x), .C0 (u, x) := C(u, x)R0 C(u, x).  .C 2 (u) := π(dx)C 2 (u, x). E .C 01 (u) := π(dx)C01 (u, x), .C01 (u, x) := C(u, x)R0 Cu (u, x). E  .C 1 (u) := E π(dx)C1 (u, x), .C1 (u, x) := C(u, x)Cu (u, x).

.

Moreover, .ϕ  and .ϕ  are the first and second derivatives of .ϕ.

4.4.4 Estimation of the Stationary Distribution We present here an application of the additive functional asymptotic results, of an estimation problem, concerning the stationary distribution .π of the SMC .(zk ). In particular, we derive asymptotic properties of this estimator. Let us observe a semi-Markov chain .(zk ) on the time interval .[0, k] ⊂ N. The empirical estimator for the stationary probability of the SMC, .π , is (see [102]) 1 1B (zs ) k k

πˆ k (B) :=

.

(4.32)

s=1

for any .B ∈ E . Let us denote by .L20 (π ) the subspace of the centred second-order elements  2  2 2 .L (π ), that is, .v ∈ L (π ) implies that . 0 E v (x)π(dx) < ∞ and . E v(x)π(dx) = 0. Consistency Under assumptions A1–A2, we have, for any fixed .B ∈ E , πˆ [t/ε] (B) ⇒ π(B),

.

ε→0

for .t ≥ 0. And, we can also state that .



lim sup πˆ [t/ε] (B) − π(B) = 0, ε↓0 0≤t≤T

for any .T ∈ (0, ∞). Asymptotic normality Let .πˆ tε (B) := πˆ [t/ε2 ] (B), .B ∈ E . Under assumptions A1– A2, we have ε−1 t[πˆ tε (B) − π(B)] ⇒ bWt ,

.

ε → 0,

4.4 Applications of the Limit Theorems to Stochastic Systems

77

where .t ≥ 0 is the standard Wiener process, and  b2 := b2 (B) := 2

π(dx)R0 (x, B) − π(B).

.

B

Hence, for an observation time of order .ε−2 , we have approximately πˆ 1ε (B) ∼ N(π(B); ε2 b2 ).

.

4.4.5 U-Statistics The U-statistical process .Um (k) in DTSMRE is given by 

Um (k) :=

a(xi1 , . . . , xim ),

.

k ∈ N,

1≤i1 0. Ej

C4: The mean merged values are positive and finite, that is,  0 < mj :=

ρj (dx)m(x) < ∞.

.

Ej

86

5 DTSMRE in Reduced Random Media

From relation (2.13), Chap. 2, we get directly πj (dx)q(x) = qj ρj (dx),

(5.7)

.

 where .q(x) := 1/m(x) and .qj := 1/mj with .mj := Ej ρj (dx)m(x). Let us consider here a separable Banach space .(B, · ) on the field of real numbers, with .B its Borel .σ -algebra, and a family of bounded contraction operators .D(x), x ∈ E, defined on .B, where the maps .D(x)ϕ : E → B are .E -measurable, .ϕ ∈ B. Denote by I the identity operator on .B. Let .Π B = N (Q ) be the null space and .(I − Π )B = R(Q ) be the range values space of operator .Q . We will  suppose here that the Markov chain .(zk , γk , k ∈ N) is uniformly ergodic, that is, .((P )n − Π )ϕ  → 0, as .n → ∞, for any .ϕ ∈ B. In that case, the transition operator is reducible-invertible on .B. Thus, we have .B = N(Q )⊕R(Q ), the direct sum of the two subspaces. Denote by .D(A) the domain of an operator A on .B. Let us consider now a DTSMRE, .Φk , k ∈ N, on .B, by (see Chap. 3) Φk ϕ = D(zk )D(zk−1 ) · · · D(z2 )D(z1 )ϕ,

.

k ≥ 1,

and

Φ0 = I,

(5.8)

for any .ϕ ∈ B0 := ∩x∈E D(D(x)). Thus we have .Φk = D(zk )Φk−1 . Consider also the average random evolution .uk (x), .x ∈ E, .k ∈ N, by uk (x) := Ex [Φk ϕ(zk )].

(5.9)

.

5.3 Average and Diffusion Approximation In this section we present average and diffusion approximation results for the discrete-time semi-Markov random evolution, as well as diffusion approximation with equilibrium.

5.3.1 Averaging Let us set .k := [t/ε], where .[x] is the integer part of the positive real number x, and consider the continuous-time process .Mtε ε Mtε := M[t/ε] = Φ[t/ε] −I −

[t/ε]−1 

.

[P D ε (·) − I ]Φε ,

t ≥ 0, ε > 0.

=0

We will prove here asymptotic results for this process, as .ε → 0.

5.3 Average and Diffusion Approximation

87

The following assumptions are needed for averaging: A1: The Markov chain .(zk , γk , k ∈ N) is uniformly ergodic in each class .Ej , with ergodic probability

πj (B × {k}),

.

B ∈ E ∩ Ej ,

k ∈ N,

and the projector operator .Π is defined by relation (5.6). A2: The moments .m2 (x), x ∈ E, are uniformly integrable. That is, (2.14), Chap. 2, holds for .r = 2. A3: Let us assume that the perturbed operator .D ε (x) has the following representation on .B: D ε (x) = I + εD1 (x) + εD0ε (x),

.

where operators .D1 (x) on .B are closed and .B0 := ∩x∈E D(D  1 (x)) isdense in B, .B 0 = B. Operators .D0ε (x) are negligible, that is, .limε→0 D0ε (x)ϕ  = 0 for .ϕ ∈ B0 .  A4: We have . Ei πi (dx) D1 (x)ϕ 2 < ∞, for .i = 1, . . . , d. A5: There exist Hilbert spaces .H and .H∗ such that compactly embedded in Banach spaces .B and .B∗ , respectively, where .B∗ is a dual space to .B. A6: Operators .D ε (z) and .(D ε )∗ (z) are contractive on Hilbert spaces .H and .H∗ , respectively. We note that if .B := C0 (R), then .H = W l,2 (R) is a Sobolev space, and l,2 .W (R) ⊂ C0 (R), and this embedding is compact (see [151]). For the spaces l,2 (R), the situation is the same. .B := L2 (R) and .H = W Theorem 5.1 Under Assumptions A1–A6 and C1–C4, the following weak convergence takes place: ε  Φ[t/ε] ⇒ Φ(t),

.

ε ↓ 0,

 is determined by the following: equation where the limit random evolution .Φ(t)  ϕ ( Φ(t) xt ) −  ϕ (u) −



.

t

  ϕ ( LΦ(s) xs )ds = 0,

0 ≤ t ≤ T,

ϕ ∈ B0 , (5.10)

0

with generator . L, defined by  LΠ = Π D1 Π + Π Q1 Π

.

(5.11)

xt is the merged Markov and acting on test functions .ϕ(x, υ(x)). The process .  with generating matrix .Q,  see Chap. 3. process, on .E,

88

5 DTSMRE in Reduced Random Media

Let us consider the average random evolution defined as  ϕ ( Λx (t) := Ex [Φ(t) x (t))],

.

x ∈ E.

1 Π = Π D1 Π and .QΠ  = Π Q1 Π . Then we have the following straightforSet .D ward result. Corollary 5.1 The average random evolution .Λx (t) satisfies the following Cauchy problem:  dΛx





dt (t) = (Q + D1 )Λx (t) ϕ (u). Λx (0) = 

.

5.3.2 Diffusion Approximation For the diffusion approximation, we will consider a different time-scaling and some additional assumptions. In this case, we replace relation (5.3) by the following one: P ε (x, B) = P (x, B) + ε2 P1 (x, B)

.

(5.12)

D1: Let us assume that the perturbed operators .D ε (x) have the following representation in .B: D ε (x) = I + εD1 (x) + ε2 D2 (x) + ε2 D0ε (x),

.

where operators .D2 (x) on B are closed and .B0 := ∩x∈E D(D2 (x)) is ε dense in .B 0 = B; operators .D (x) are a negligible operator, that is, 0   .B, ε   .limε↓0 D (x)ϕ = 0. 0 D2: The following balance condition holds: Π D1 (x)Π = 0.

.

(5.13)

D3: The moments .m3 (x), x ∈ E, are uniformly integrable, that is, relation (2.14), Chap. 2, holds for .r = 3. Theorem 5.2 Under Assumptions A1, A5–A6 (see Sect. 3.1), and D1–D3, the following weak convergence takes place: ε Φ[t/ε 2 ] ⇒ Φ0 (t),

.

ε ↓ 0,

5.4 Proof of Theorems

89

where the limit random evolution .Φ0 (t) is a diffusion random evolution determined  by the following generator .L,   LΠ = Π D2 Π + Π D1 R0 D1 Π − Π D12 Π + QΠ

.

 is the generating matrix of the merged Markov process .( where .Q xt ), see Chap. 3.

5.3.3 Normal Deviations We note that averaged semi-Markov random evolutions can be considered as the first approximation to the initial evolutions. The diffusion approximation of the semi-Markov random evolutions determines the second approximation to the initial evolution, since the first approximation under balance condition appears to be trivial. Here we consider the algorithms of construction of the first and the second approximation in the case when the balance condition in the diffusion approximation scheme is not fulfilled. We introduce the deviated semi-Markov random evolution as the normalized difference between the initial and averaged evolutions. In the limit, we obtain the diffusion approximation with equilibrium of the initial evolution from the averaged one. ε Let us consider the discrete-time semi-Markov random evolution .Φ[t/ε] , aver (see Sect. 3.1), and the deviated evolution aged evolution .Φ(t) ε  Wtε := ε−1/2 [Φ[t/ε] − Φ(t)].

.

(5.14)

Theorem 5.3 Under Assumptions A1, A5–A6 (see Sect. 5.3.1), and D3, the deviated semi-Markov random evolution .Wtε weakly convergence, when .ε → 0, to the diffusion random evolution .Wt0 defined by the following generator:  1 )R0 (D1 − D 1 )Π + QΠ,  LΠ = Π (D1 − D

.

1 is the contracted operator .D1 as given in Sect. 5.3.1, and .Q  is the where .D generating matrix of the merged Markov process .( xt ), see Chap. 3.

5.4 Proof of Theorems  of the switching SMC .(zk ) is a finite set, we do not consider As the state space .E the new component .υ(zk ), and the proof of tightness in Chap. 3 is also valuable here. So, we will only prove here the finite dimensional distributions convergence concerned by the transition kernels.

90

5 DTSMRE in Reduced Random Media

5.4.1 Proof of Theorem 5.1 The perturbed semi-Markov kernel .q ε has the representation q ε (x, B, k) = q(x, B, k) + εq1 (x, B, k),

.

where .q1 (x, dy, k) := P1 (x, dy)fx (k). The discrete generators of the four-component family of processes ε Φ[t/ε] ϕ,

.

ε z[t/ε] ,

ε υ(z[t/ε] ),

ε γ[t/ε] ,

t ≥ 0, ε > 0,

are L ϕ(u, x, υ(x), k) = ε ε

.

−1

 P (x, k; dy, k + 1)[D ε (y)ϕ(u, y; υ(y), k + 1) E

−ϕ(u, x, υ(x), k)].

(5.15)

The asymptotic representation of the above operator acting on test functions ϕ(u, x, υ(x), k) is given by

.

Lε ϕ(u, x, υ(x), k) = [ε−1 Q ,ε + P ,ε D1 (·) + P ,ε D0ε (·)]ϕ(u, x, υ(x), k), (5.16)

.

where .Q ,ε := P ,ε − I . Now, from (5.3), the transition operator .P ,ε can be written as follows: P ,ε = P + εQ1 ,

.

where the operator .Q1 is defined by relation (see also Chap. 2)  Q1 ϕ(x, υ(x)) = q(x)

P1 (x, dy)ϕ(y, υ(y)),

.

(5.17)

E

where .q(x) := 1/m(x), .x ∈ E. Finally, the asymptotic representation of the operator .Lε can be written as Lε (x) = ε−1 Q + Q1 + P D1 (x) + θ ε (x),

.

where .Q := P − I , and the negligible operator .θ ε (x) is given by θ ε (x) := P ,ε D0ε (x) + εQ1 (D1 (x) + D0ε (x)].

.

(5.18)

5.5 Application to Stochastic Systems

91

From the singular perturbation problem .Lε ϕ(u, x) = Lϕ(υ(x))+θ ε (x), with test LΠ = Π (P D1 (x) + functions .ϕ ε = ϕ + εϕ1 , we get the limit operator defined by . Q1 )Π (see Proposition 1.1 in Chap. 1 and also Proposition 5.1 in [90]). From this representation, we get  1 + Q,  L=D

.

1 Π = Π D1 (x)Π and .QΠ  = Π Q1 Π . where .D



.

5.4.2 Proof of Theorem 5.2 The discrete generators of the four-component family of processes ε Φ[t/ε 2 ] ϕ,

ε z[t/ε 2],

.

ε υ(z[t/ε 2 ] ),

ε γ[t/ε 2],

t ≥ 0, ε > 0,

are Lε (x) = ε−2 Q + ε−1 P D1 (x) + Q1 + P D2 (x)Θ ε (x).

.

(5.19)

Solving singular perturbation problem Lε (x)ϕ(x, k) = Lϕ(υ(x)) + θ ε (x, k),

.

with test functions .ϕ ε = ϕ + εϕ1 + ε2 ϕ2 (see Proposition 1.2 in Chap. 1, and Proposition 5.2 in [90]), we get the desired result. . The proof of Theorem 5.3 is similar to the previous ones.

5.5 Application to Stochastic Systems We will apply results of the previous sections to special stochastic systems, as additive functionals, dynamical systems, U-statistics, etc. The merged SMC on the split state space will be the same as in Sect. 5.2. The proofs of Propositions 5.1 to 5.12 below are obtained directly as corollaries from Theorems 5.1 to 5.3.

92

5 DTSMRE in Reduced Random Media

5.5.1 Additive Functionals The integral functional of a semi-Markov chain considered here is defined by yk := u +

k 

.

k ≥ 1,

a(zs ),

y0 = u,

s=1

where a is a real-valued measurable function defined on the state space E. The perturbing operator .D ε (x) is defined as follows: D ε (x)ϕ(u) = ϕ(u + εa(x)).

.

The perturbed operator .D ε (x) has the asymptotic expansion (5.13) with 1   .D1 (x)ϕ(u) = a(x)ϕ (u) and .D2 (x)ϕ(u) = a(x)ϕ (u). 2 Average approximation In the averaging scheme, the additive functional has the representation ytε := ε

[t/ε] 

.

t ≥ 0, ε > 0,

a(zs ),

y0ε = u.

(5.20)

s=1

Then, from Theorem 5.1, we obtain the following proposition: Proposition 5.1 Under conditions C1–C4 and A1–A2, the following weak convergence holds: ytε ⇒  yt ,

as

.

ε → 0,

where the limit process is an integral functional, defined by   yt :=

.

t

 a ( xs )ds,

0

 with  .a (j ) = xt is defined on the state space Ej πj (dx)a(x). The Markov process .   .E as in the previous section by the generator .Q. Diffusion approximation In the diffusion approximation, the additive functional is ε .ξt

:= ε

2 [t/ε ]

s=1

a(zs ),

t ≥ 0, ε > 0,

ξ0ε = u.

(5.21)

5.5 Application to Stochastic Systems

93

Then, from Theorem 5.2, we obtain the following proposition: Proposition 5.2 Under conditions C1–C4, A1–A2, and D2, the following weak convergence holds: ξtε ⇒ ξt0 ,

.

asε → 0,

where the limit process is a diffusion process, dξt0 = b( xt )dwt .

.

a2 (j ), The process .wt , t ≥ 0, is a standard Brownian motion, and .b2 (j ) :=  a0 (j )− 12  where    .a0 (j ) := πj (dx)a(x)R0 a(x), and  a2 (j ) := πj (dx)a 2 (x). Ej

Ej

Remark 5.1 The generator of the switched diffusion .ξt0 is given by 1 2 b (j ) ϕj (u)1j (x). 2 d

Lϕ(u, x) =

.

(5.22)

j =1

Normal deviations The normal deviations, or diffusion approximation with equilibrium, will be realized without balance condition D2. Let us consider the stochastic processes .ζtε , t ≥ 0, ε > 0, ζtε := ε−1/2 (ytε −  yt ).

.

The process . yt is the limit process in the averaging scheme. Then, from Theorem 5.3, we obtain the following proposition: Proposition 5.3 Under conditions C1–C4 and A1–A2, the following weak convergence holds: ζtε ⇒ ζt0 ,

.

as ε → 0,

where the limit process is a diffusion process, dζ0 (t) = c( xt )dwt .

.

The process .wt , t ≥ 0, is a standard Brownian motion, and  c2 (j ) :=

πj (dx)( a (j ) − a(x))R0 ( a (j ) − a(x)),

.

Ej

1 ≤ j ≤ d.

94

5 DTSMRE in Reduced Random Media

5.5.2 Dynamical Systems Let us consider the difference equation ε yk+1 = ykε + εC(ykε ; zk+1 ),

.

k ≥ 0,

and

y0ε = u,

(5.23)

switched by the SMC .(zk ). The perturbed operators .D ε (x), x ∈ E, are defined now by D ε (x)ϕ(u) = ϕ(u + εC(u, x)).

.

The perturbed operator .D ε (x) has the asymptotic expansion in A3 with 1   .D1 (x)ϕ(u) = C(u, x)ϕ u) and .D2 (x)ϕ(u, x) = C(u, x)ϕ (u). 2 Average approximation The time-scaled system considered here is ε ε ε ytε := y[t/ε]+1 = y[t/ε] + εC(y[t/ε] ; z[t/ε]+1 ),

.

t ≥ 0,

and

y0ε = u.

(5.24)

Then, from Theorem 5.1, we obtain the following proposition: Proposition 5.4 Under conditions C1–C4 and A1–A2, the following weak convergence holds: ytε ⇒  yt ,

.

as ε → 0,

where the limit process is a continuous-time dynamical system, defined by .

 j) = with .C(u,

 Ej

d  yt ;  xt ),  yt := C( dt

πj (dx)C(u, x).

Diffusion approximation The time-scaled dynamical system considered here is ε ε ε ξtε := y[t/ε 2 ]+1 = y[t/ε 2 ] + εC(y[t/ε 2 ] ; z[t/ε 2 ]+1 ),

.

t ≥ 0,

and

y0ε = u. (5.25)

Then, from Theorem 5.2, we obtain the following proposition: Proposition 5.5 Under conditions C1–C4, A1–A2, and D2, the following weak convergence holds: ξtε ⇒ ξt0 ,

.

as ε → 0,

where the limit process is a diffusion process, dξt0 = c( xt )dwt .

.

5.5 Application to Stochastic Systems

95

The process .wt , t ≥ 0, is a standard Brownian motion, and 1 2 (u, j ). 0 (u, j ) − C C(u, j ) + C 2

c2 (j ) :=

.

The coefficients are, for .j = 1, 2, . . . , d, 

 j ) := .C(u,

πj (dx)C(u, x),

0 (u, j ) := C

Ej

 πj (dx)C(u, x)R0 C(u, x), Ej

2 (u, j ) := C

and

.

 πj (dx)C 2 (u, x). Ej

Normal Deviations The time-scaled system considered for the normal deviations, or diffusion approximation with equilibrium, is ζtε := ε−1/2 (y[t/ε] −  yt ),

.

t ≥ 0,

and

ζ0ε = u.

(5.26)

Then, from Theorem 5.3, we obtain the following proposition: Proposition 5.6 Under conditions C1–C4 and A1–A2, the following weak convergence holds: ζtε ⇒ ζt0 ,

.

as ε → 0,

where the limit process is a diffusion process, dζt0 = ce ( xt )dwt .

.

The process .wt , t ≥ 0, is a standard Brownian motion, and  2 .ce (j )

:=

 j )]R0 [C(u, x) − C(u,  j )]. πj (dx)[C(u, x) − C(u,

Ej

5.5.3 Geometric Markov Renewal Chains The GMRC introduced in the previous chapter can be studied in the reduced media case as follows. Averaging Define the following sequence of processes: Stε := S0ε

[t/ε] 

(1 + εa(zk )),

.

k=1

t ∈ R+ ,

S0ε = s,

ε > 0.

96

5 DTSMRE in Reduced Random Media

Then, from Theorem 5.1, we obtain the following proposition: Proposition 5.7 Under averaging conditions, the above process .Stε weakly converges to process .S¯t x (t))t , S¯t = sea (

.

where  .a (j ) :=



πj (dz)a(z), for .j = 1, 2, . . . , d.

Ej

Diffusion Approximation Define the following sequence of processes: S (t) := S0

.

ε

2 [t/ε ]

(1 + εa(zk )),

t ∈ R+ ,

S0ε = s.

k=1

Then, from Theorem 5.2, we obtain the following proposition: Proposition 5.8 The above process .S(t)ε converges weakly to process .S0 (t), given by x (t))/2 σa ( S0 (t) = se−ta2 ( e x (t))w(t) ,

.

where   .a2 (j ) :=

πj (dz)a 2 (z), Ej

 σa2 (j ) :=

πj (dz)[a 2 (z)/2 + a(z)R0 a(z)],

.

Ej

where .R0 is the potential of the SMC .(zk ), and .j = 1, 2, . . . , d. It means that .S0 (t) satisfies the following stochastic differential equation, switched by the Markov process . x (t): dS0 (t) =

.

1 2 (σ ( x (t)) −  a2 ( x (t)))S0 (t)dt + σa ( x (t))S0 (t)dwt , 2 a

where .(wt ) is a standard Wiener process. Normal Deviations Let us consider the following normalized GMRC: a t], S˜tε := ε− 2 [ln(Stε /S0 ) −  1

.

t ≥ 0,

S0ε = s.

5.5 Application to Stochastic Systems

97

Then, from Theorem 5.3, we obtain the following proposition: Proposition 5.9 The process .S˜tε converges weakly to the process .σ wt , where  σ (j ) :=

.

πj (a(z) −  a (j ))R0 (a(z) −  a (j )),

2

j = 1, 2, . . . j = 1, 2, . . . , d,

Ej

and .wt is a standard Wiener process. In this way, the GMRC .S˜tε may be presented in the following approximated form: √ εσ ( x (t))wt

x (t))t+ S˜tε ≈ S0 ea (

.

.

5.5.4 U-Statistics Averaging with merging Then the U-process (4.33), for .a(xi , xj ) = 0, if .j = i + 1, is ε ν[t/ε]

ε .U2 (t)

:= ε



t ≥ 0,

a(xi−1 , xi ),

ε > 0.

(5.27)

i=1

The random evolution can be written as Φ ε ϕ(u) = ϕ(u + U2ε (t)),

.

t ≥ 0,

and the family of operators D ε (x, y)ϕ(u) := ϕ(u + εa(x, y)),

.

for .ϕ ∈ C0 (R). We assume that operator .D ε (x, y) has the asymptotic representation D ε (x, y) = I + εD(x, y) + εD0 (x, y),

.

where .D(x, y)ϕ(u) = a(x, y)ϕ  (u), for .ϕ ∈ C 1 (R), and .D0ε (x, y) a negligible operator. And the common domain of operators .D(x, y), .x, y ∈ E, is the space 1 .C (R) independently of .x, y. Then, from Theorem 5.1, we obtain the following proposition: Proposition 5.10 The following weak convergence holds, when ε → 0: (t) := U2ε (t) ⇒ U



t

.

0

 a ( x (s))ds,

98

5 DTSMRE in Reduced Random Media

where 



 .a (j ) :=

ρj (dx)

P (x, dy)a(x, y)/mj

Ej

and mj :=

 Ej

Ej

 ρj (dx)m(x), j ∈ E.

Diffusion approximation with merging Let us consider the following family of U-processes: νε

U2ε (t) := ε

[t/ε ]  2

(5.28)

a(xi−1 , xi ).

.

i=1

We assume that operator D ε (x, y) has the asymptotic representation D ε (x, y) = I + εD1 (x, y) + ε2 D2 (x, y) + ε2 D0ε (x, y),

.

where D1 (x, y)ϕ(u) = a(x, y)ϕ  (u), and D2 (x, y)ϕ(u) = a(x, y)ϕ  (u)/2 for ϕ ∈ C 2 (R). And the domain of operators D(x, y), x, y ∈ E, is the space C 1 (R) independently of x, y. Then, from Theorem 5.2, we obtain the following proposition: Proposition 5.11 The following weak convergence holds, when ε → 0:  ε .U2 (t)

t

⇒ U (t) := 0

 σ ( x (s))ds,

0

where 



 σ 2 (j ) :=

ρj (dx)

.

Ej

+

1 2

 P (x, dy)a(x, y)R0

Ej



P (x, dy)a(x, y) Ej

P (x, dy)a 2 (x, y) /mj , Ej

for j = 1, 2, . . . , d. Normal deviations Let us consider the U-process in the same series scheme as in the averaging, that is, ν[t/ε]

U2ε (t) := ε



.

i=1

a(xi−1 , xi ).

(5.29)

5.6 Concluding Remarks

99

Define now the process 2 (t)]. U ε (t) := ε−1/2 [U2ε (t) − U

.

(5.30)

Then, from Theorem 5.3, we obtain the following proposition: Proposition 5.12 The following weak convergence holds, when ε → 0: 

t

U (t) ⇒ U (t) := 0

ε

.

 σ ( x (s))ds

0

where 

 ρj (dx) (

 σ (j ) :=

.

2

Ej



P (x, dy)a(x, y) −  a (j ))

( Ej

1 + ( 2 and mj :=

 Ej

P (x, dy)a(x, y) −  a (j ))(R0 − I )

Ej



P (x, dy)a 2 (x, y) −  a (j )) /mj .

Ej

 ρj (dx)m(x), j ∈ E,   .a (j ) :=

 ρj (dx)

Ej

P (x, dy)a(x, y)/mj . E

In regard to the results in this section, it is worth noticing that the limit processes, as the original ones, are also switching processes but with an important difference that the limit switching processes are much simpler. For example, from a family of semi-Markov chains with general state space, we get as limit Markov chains with finite state space in continuous-time. Moreover we can also consider mixed levels of time-scaling which allow one to consider double merging scheme, etc., see, e.g., [90–92].

5.6 Concluding Remarks In this chapter, we presented results for reducing random media. These results are related also to the merging approach in Chap. 2. The reduction of random media is an important tool, since it simplifies considerably the very complex systems. Also, as in the merging phase space in Chap. 2, it can be regarded in several ways, e.g., double merging in random media. Rates of convergence can be obtained in the same way applying random evolution results.

Chapter 6

Controlled Discrete-Time Semi-Markov Random Evolutions

6.1 Introduction In this chapter, we introduced controlled discrete-time semi-Markov random evolutions. These processes are random evolutions of discrete-time semi-Markov processes where we consider a control applied to the values of random evolution. The main results concern time-rescaled weak convergence limit theorems in a Banach space of the above stochastic systems as averaging and diffusion approximation. The applications are given to the controlled additive functionals, controlled geometric Markov renewal processes, and controlled dynamical systems. We provide dynamical principles for discrete-time dynamical systems such as controlled additive functionals and controlled geometric Markov renewal processes. We also produce dynamic programming equations (Hamilton–Jacobi–Bellman equations) for the limiting processes in diffusion approximation such as controlled additive functionals, controlled geometric Markov renewal processes, and controlled dynamical systems. As an example, we consider the solution of portfolio optimization problem by Merton for the limiting controlled geometric Markov renewal processes in diffusion approximation scheme. The rates of convergence in the limit theorems are also presented. Compared to the previous chapters, here we considered additionally a control on the random evolution, which we call controlled discrete-time semi-Markov random evolution (CDTSMRE) in a Banach space, and we presented time-rescaled convergence theorems. In particular, we get weak convergence theorems in Skorokhod space .D[0, ∞) for càdlàg stochastic processes, see, e.g., in [73]. The limit theorems include averaging, diffusion approximation, and diffusion approximation with equilibrium. For the above limit theorems, we also presented rates of convergence results. Finally, we give some applications regarding the above mentioned results, especially to controlled additive functionals (CAFs), CGMRP, and controlled dynamical systems (CDSs), and optimization problems. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0_6

101

102

6 Controlled Discrete-Time Semi-Markov Random Evolutions

Regarding optimization issues, we provide dynamical principles for discretetime dynamical systems such as CAF and CGMRPs (see Sect. 6.2.3), see, e.g., [13, 48, 95]. We also produce dynamic programming equations (Hamilton–Jacobi– Bellman equations) for the limiting processes in diffusion approximation such as CAF, CGMRP, and CDS. As an example, we consider the solution of portfolio optimization problem by Merton for the limiting CGMRP in DA (see Sect. 6.4.4). Merton problem, or Merton portfolio’s problem, is a problem in continuous-time finance associated with portfolio choice. In .(B, S)-security market, which consists of a stock and a risk-free asset, an investor must choose how much to consume and must allocate his wealth between the stock and the risk-free asset in such a way that maximizes expected utility. The problem was formulated and first solved by Robert Merton in 1969 and published in 1971 [112]. This chapter contains new and original results on dynamical principle concerning CDTSMRE and DPE (HJB equations) for the limiting processes in DA. One of the new remarkable results is the solution of Merton portfolio problem for the limiting CGMRP in DA. The method of proofs was based on the martingale approach together with convergence of transition operators of the extended semiMarkov chain via a solution of a singular perturbation problem [90, 91, 154]. As in our previous work [104], the tightness of these processes is proved via Sobolev’s embedding theorems [2, 139, 151]. It is worth mentioning that, as in the Markov case, the results presented here cannot be deduced directly from the continuoustime case. We should also note that DTSMREs have been completely studied in [104]. For semi-Markov processes, see, e.g., [132–134, 143]. For Markov chains and additive functionals, see, e.g., [111, 120, 136, 146, 148]. The chapter is organized as follows. Definition and properties of discrete-time semi-Markov random evolutions and Controlled DTSMREs, as well as particular stochastic systems as applications, are introduced in Sect. 6.2. The main results of this chapter, limit theorems of CDTSMRE, as averaging, diffusion approximation and diffusion approximation with equilibrium of controlled DTSMREs are considered in Sect. 6.3. In Sect. 6.4, we provide three applications of averaging, diffusion approximation, and diffusion approximation with equilibrium of controlled DTSMREs: controlled additive functionals, controlled GMRP, and controlled dynamical systems. Section 6.6 deals with the analysis of the rates of convergence in the limit theorems, presented in the previous sections, for controlled DTSMREs and for CAF and CGMRP. In Sect. 6.7, we give the proofs of theorems presented in the previous sections. The last section concludes the chapter and indicates some future works.

6.2 Controlled Discrete-Time Semi-Markov Random Evolutions

103

6.2 Controlled Discrete-Time Semi-Markov Random Evolutions 6.2.1 Definition of CDTSMREs We define here controlled discrete-time semi-Markov random evolutions. Let U denote a compact Polish space representing the control, and let .uk be U -valued control process and we suppose that it is a Markov chain. We note that we could also define the process .uνk , which is a semi-Markov control process, considered in many papers (see, e.g., in [72, 172]). We suppose that homogeneous Markov chain u = P(u .uk is independent of .zk and transition probability kernel .P k+1 ∈ dy | uk = u) = Q(u, dy). Let us consider a family of bounded contraction operators .D(z, u), z ∈ E, u ∈ U , defined on .B, where the maps .D(z, u)ϕ : E × U → B are .E × U -measurable,  .ϕ ∈ B. Denote by I the identity operator on B. Let .Π B = N (Q ) be the null  space and .(I − Π )B = R(Q ) be the range value space of operator .Q . We will suppose here that  the Markov chain .(zk , γk , k ∈ N) is uniformly ergodic, that is,   n .((P ) − Π )ϕ  → 0, as .n → ∞, for any .ϕ ∈ B. In that case, the transition operator is reducible-invertible on .B. Thus, we have .B = N (Q ) ⊕ R(Q ), the direct sum of the two subspaces. The domain of an operator A on .B is denoted by .D(A) := {ϕ ∈ B : Aϕ ∈ B}. Let us define here, as in Chap. 3, the random evolution of SMC but with the additional component, .uk , concerning the control. Definition 6.1 A controlled discrete-time semi-Markov random evolution (CDTSMRE) .(Φku , k ∈ N) on the Banach space .B, is defined by Φku ϕ = D(zk , uk )D(zk−1 , uk−1 ) · · · D(z2 , u2 )D(z1 , u1 )ϕ,

.

(6.1)

for .k ≥ 1, Φ0u = I, u0 = u ∈ U , and for any .ϕ ∈ B0 := ∩x∈E,u∈U D(D(x, u)). Thus we have .Φk = D(zk , uk )Φk−1 . The process .(zk , γk , uk ) is a Markov chain on .E × N × U , adapted to the filtration Fku := σ (z , γ , u ; ≤ k), .k ≥ 0. We also note that .(Φku ϕ, zk , γk , uk ) is a Markov chain on .B × E × N × U with discrete generator

.

+ .L ϕ = [P

 

(·, dv)(D(v, u) − I )]ϕ, P

u

E

U

where .ϕ := ϕ(x, z, s, u), and ϕ(z, s, u) := P  P u ϕ(z, s, u) = .P

 s ∈N

×ϕ(z , s , u ).

P  (z, s; dz , s )P u (u; du ) E×U

(6.2)

104

6 Controlled Discrete-Time Semi-Markov Random Evolutions

The process .Mku defined by Mku := Φku − I −

k−1 

.

u E[Φ +1 − Φ u | F u ],

k ≥ 1,

M0 = 0,

(6.3)

=0

on .B, is an .Fku -martingale. The random evolution .Φku can be written as follows: Φku := I +

.

k−1  [D(z +1 , u +1 ) − I ]Φ u , =0

and then, the martingale (6.3) can be written as follows: u .Mk

:=

Φku

−I −

k−1 

E[(D(z +1 , u +1 ) − I )Φ u | F u ],

=0

or Mku := Φku − I −

k−1 

.

[E(D(z +1 , u +1 ) | F u ) − I ]Φ u .

=0

D(·)Φ ϕ](z , γ , u ), one takes Finally, as .E[D(z +1 , u +1 )Φ u ϕ | F u ] = [P Mku := Φku − I −

k−1 

.

D(·, u) − I ]Φ u . [P

=0

6.2.2 Examples Let us give here some typical examples of CDTSMRE encountered in theory and practice of stochastic processes. Example 6.1 (Controlled Additive Functional or Markov Decision Process) Let us define the following controlled additive Functional: yku =

k 

.

a(zl , ul ),

k ≥ 0,

y0 = y.

l=0

If we define the operator .D(z, u) on .C0 (R) in the following way: D(z, u)ϕ(y) := ϕ(y + a(z, u)),

.

6.2 Controlled Discrete-Time Semi-Markov Random Evolutions

105

then the controlled discrete-time semi-Markov random evolution .Φk ϕ has the following presentation, Φku ϕ(y) = ϕ(yku ).

.

Process .yku is usually called in the literature the Markov decision process (see, e.g., in [3, 14, 20, 23]). Example 6.2 (Controlled Geometric Markov Renewal Process) The CGMRP is defined in the following way: Sku := S0

k 

.

(1 + a(zl , ul )),

k ∈ N,

S0 = s.

l=1

 We suppose that . 0k=1 = 1. If we define the operator .D(z, u) on .C0 (R) in the following way: D(z, u)ϕ(s) := ϕ(s(1 + a(z, u))),

.

then the controlled discrete-time semi-Markov random evolution .Φku ϕ can be given as follows: Φku ϕ(s) = ϕ(Sku ).

.

To the authors’ opinion, this process is defined for the first time in the literature and the notion of controlled GMRP is a new one as well.

6.2.3 Dynamic Programming for Controlled Models Here, we present dynamic programming for controlled models given in Examples in the previous section. Let us consider a Markov control model (see [63]) .(E, A, {A(z)|z ∈ E}, Q, c). Here, E is the state space; A is the control or action set; Q is the transition kernel, i.e., a stochastic kernel on E given .K, where .K := {(z, u)|z ∈ E, u ∈ A(z)}; and .c : K → R is a measurable function called the cost-per-stage function. We are interested in minimizing the finite-horizon performance criterion either (see Example 6.1) J1 (π, z) := Eπz [

N −1 

.

l=0

a(zl , ul ) + aN (zN )]

106

6 Controlled Discrete-Time Semi-Markov Random Evolutions

or (see Example 6.2) J2 (π, z) := Eπz [ln(

k 

.

(1 + a(zl , ul ))(1 + aN (zN )))],

l=1

where .aN (zN ) is the terminal cost function and .π ∈ Π is the set of control policies. In this way, denoting by .J ∗ the value function Ji∗ (z) := inf Ji (π, z),

.

Π

z ∈ E,

i = 1, 2,

the problem is to find a policy .π ∗ ∈ Π such that Ji (π ∗ , z) = Ji∗ (z),

z ∈ E,

.

i = 1, 2.

Example 6.3 (Controlled Additive Functional) Let us provide an algorithm for finding both the value function .J ∗ and an optimal policy .π ∗ for the example with function .J1 (π, z) (see Example 6.1). Let .J0,1 , J1,1 , . . . , JN,1 be the functions on E defined from .l = N to .l = 0 by (backward) JN,1 (z) := aN (z),

.

l=N

and  Jl,1 (z) := min[a(z, u) +

Jl+1,1 (y)Q(u, dy)],

.

A(z)

l = N − 1, N − 2, . . . , 0.

E

Suppose that there is a selector .ft ∈ F such that .fl (z) ∈ A(z) attains the minimum in the above expression for .Jl (z) for all .z ∈ E, meaning for any .z ∈ E and .l = 0, . . . , N − 1,  .Jl,1 (z) = a(z, fl ) + Jl+1,1 (y)Q(fl , dy). E

Then, the optimal policy is the deterministic Markov one .π ∗ = {f0 , . . . , fN −1 }, and the value function .J ∗ equals .J0 , i.e., J1∗ (z) = J0 (z) = J1 (π ∗ , z),

.

z ∈ E.

Example 6.4 (Controlled Geometric Markov Renewal Chain) Let us provide an algorithm for finding both the value function .J ∗ and an optimal policy .π ∗ for the example with function .J2 (π, z) (see Example 6.2). We will modify the expression

6.3 Limit Theorems for Controlled Semi-Markov Random Evolutions

107

Su

for .Sku in Example 6.2. Let .ln( Sk0 ) be a log-return, then  Sku )= ln(1 + a(zl , ul ). S0 k

.

ln(

l=1

Thus, we are interested in minimizing the finite-horizon performance criterion for J2 (π, z) := Eπz [

N −1 

.

ln(1 + a(zl , ul ) + ln(1 + aN (zN )].

l=0

Let .J0,2 , J1,2 , . . . , JN,2 be the functions on E defined from .l = N to .l = 0 by (backward) JN,2 (z) := ln(1 + aN (z)),

.

l=N

and  Jl,2 (z) := min[ln(1+a(z, u))+

Jl+1,1 (y)Q(u, dy)],

.

A(z)

l = N −1, N −2, . . . , 0.

E

Suppose that there is a selector .ft ∈ F such that .fl (z) ∈ A(z) attains the minimum in the above expression for .Jl (z) for all .z ∈ E, meaning for any .z ∈ E and .l = 0, . . . , N − 1,  .Jl,2 (z) = ln(1 + a(z, fl )) + Jl+1,2 (y)Q(fl , dy|z). E

Then, the deterministic Markov policy .π ∗ = {f0 , . . . , fN −1 } is optimal, and the value function .J ∗ equals .J0 , i.e., J2∗ (z) = J0 (z) = J2 (π ∗ , z),

.

z ∈ E.

6.3 Limit Theorems for Controlled Semi-Markov Random Evolutions In this section, we present averaging, diffusion approximation, and diffusion approximation with equilibrium results for the controlled discrete-time semi-Markov random evolutions. It is worth noticing that the main scheme of results is almost the same as in our previous works in particular [104]. Nevertheless, the additional component of the control allows us to study more interesting problems.

108

6 Controlled Discrete-Time Semi-Markov Random Evolutions

6.3.1 Averaging of CDTSMREs We consider here CDTSMREs defined in Sect. 6.2. Let us now set .k := [t/ε] and consider the continuous-time process .Mtε ε,u u Mtε,u := M[t/ε] = Φ[t/ε] −I −

[t/ε]−1 

.

D ε (·, u) − I ]Φ ε,u . [P

=0

We will prove here asymptotic results for this process as .ε → 0. The following assumptions are needed for averaging: A1: The MC .(zk , γk , k ∈ N) is uniformly ergodic with ergodic distribution .π  (B × {k}), B ∈ E , k ∈ N. A2: The moments .m2 (x), x ∈ E, are uniformly integrable. A3: The perturbed operators .D ε (x) have the following representation on .B: D ε (x, u) = I + εD1 (x, u) + εD0ε (x, u),

.

A4: A5: A6: A7:

where operators .D1 (x, u) on .B are closed and .B0 := ∩x∈E,u∈U D(D1 (x, u)) is dense .B, .B0 = B. Operators .D0ε (x, u) are negligible, i.e.,   in ε   .limε→0 D (x, u)ϕ = 0 for any .ϕ ∈ B0 .  0 We have . E U [π(dx)π1 (du)] D1 (x, u)ϕ2 < ∞. (See A7.) There exist Hilbert spaces H and .H ∗ such that compactly embedded in Banach spaces .B and .B∗ , respectively, where .B ∗ is a dual space to .B. Operators .D ε (x) and .(D ε )∗ (x) are contractive on Hilbert spaces H and .H ∗ , respectively. The MC .(uk , k ∈ N) is independent of .(zk ) and is uniformly ergodic with stationary distribution .π1 (du), k ∈ N.

We note that if .B = C0 (R), then .H = W l,2 (R) is a Sobolev space, and ⊂ C0 (R) and this embedding is compact (see [151]). For the spaces l,2 (R), the situation is the same. .B = L2 (R) and .H = W We also note that semi-Markov chain .(zk , uk ) is uniformly ergodic on .E × U with stationary probabilities .π(dx)π1 (du), which follows from conditions A1 and A7. l,2 (R) .W

Theorem 6.1 Under Assumptions A1–A7, the following weak convergence takes place: ε Φ[t/ε,u]

⇒ Φ(t),

.

ε ↓ 0,

where the limit random evolution .Φ(t) is determined by the following equation: 

t

Φ(t)ϕ − ϕ −

.

0

LΦ(s)ϕds = 0,

0 ≤ t ≤ T,

ϕ ∈ B0 ,

(6.4)

6.3 Limit Theorems for Controlled Semi-Markov Random Evolutions

109

or, equivalently, .

d Φ(t)ϕ = LΦ(t)ϕ, dt

where the limit contracted operator is then given by 1 = L=D .

  [π(dx)π1 (du)]D1 (x, u). E

(6.5)

U

This result generalizes the classical Krylov–Bogolyubov averaging principle [93] on a Banach space and a control space.

6.3.2 Diffusion Approximation of DTSMREs For the diffusion approximation of CDTSMREs, we will consider a different timescaling and some additional assumptions. D1: Let us assume that the perturbed operators .D ε (x, u) have the following representation in .B: D ε (x, u) = I + εD1 (x, u) + ε2 D2 (x, u) + ε2 D0ε (x, u),

.

where operators .D2 (x, u) on .B are closed and .B0 := ∩x∈E,u∈U D(D2 (x, u)) ε is dense  inε .B, .B 0 = B; operators .D0 (x, u) are a negligible operator, i.e., .limε↓0 D (x, u)ϕ  = 0. 0 D2: The following balance condition holds: Π D1 (x, u)Π = 0,

.

(6.6)

where  

Π ϕ(x, k, u) :=

π  (dy × )π1 (du)ϕ(y, , u)1(x, k).

.

l≥0

E

(6.7)

U

D3: The moments .m3 (x), x ∈ E, are uniformly integrable. Theorem 6.2 Under Assumptions A1, A5–A7 (see Sect. 6.3.1), and D1–D3, the following weak convergence takes place: ε Φ[t/ε 2 ] ⇒ Φ0 (t),

.

ε ↓ 0,

110

6 Controlled Discrete-Time Semi-Markov Random Evolutions

where the limit random evolution .Φ0 (t) is a diffusion random evolution determined by the following generator: L = Π D2 (x)Π + Π D1 (x)R0 D1 (x)Π − Π D12 (x)Π,

.

where  + Π ]−1 − Π, R0 := [Q

(6.8)

 := P  − I. Q

(6.9)

.

and .

6.3.3 Normal Approximation The diffusion approximation with equilibrium or the normal deviations is obtained by considering the difference between the rescaled initial processes and the averaging limit process. This is of great interest when we have no balance condition as previously in the standard diffusion approximation scheme. Consider now the controlled discrete-time semi-Markov random evolution ε .Φ [t/ε,u] , averaged evolution .Φ(t) (see Sect. 6.3.1) and the deviated evolution ε,u Wtε,u := ε−1/2 [Φ[t/ε] − Φ(t)].

.

(6.10)

Theorem 6.3 Under Assumptions A1, A5–A6 (see Sect. 6.3.1), and D3, with operators .D ε (x) in A3, instead of D1, the deviated controlled semi-Markov random evolution .Wtε,u weakly convergence, when .ε → 0, to the diffusion random evolution 0 .Wt defined by the following generator: L = Π (D1 (x, u) − D 1 )R0 (D1 (x, u) − D 1 )Π,

.

(6.11)

where .Π is defined in (6.7).

6.4 Applications to Stochastic Systems In this section, in connection with the above results, we present two controlled stochastic systems: controlled additive functionals, which have many applications, e.g., in storage, reliability, and risk theories (see, e.g., in [12, 90, 91, 103]), and controlled geometric Markov renewal processes, which also have many applications, particularly in finance (see [159–161, 163]). Our main goal here is to get the

6.4 Applications to Stochastic Systems

111

limiting processes and apply optimal control methods to receive the solutions of optimization problems. The limiting results for MC such as LLN and CLT were considered in [4, 5].

6.4.1 Controlled Additive Functionals Let us consider here the CAF, .(yku ), described previously in Example 6.1. Averaging of CAF Now, if we define the continuous time process ytε,u := ε

[t/ε] 

.

a(zl , ul ),

l=0

then from Theorem 6.1 it follows that this process has the following limit .y0 (t) = limε→0 ytε : y0 (t) = y + at, ˆ

.

    where .aˆ = E U π(dz)π1 (du)a(z, u). We suppose that . E U π(dz)π1 (du) .|a(z, u)| < +∞. Diffusion Approximation of CAF If we consider the continuous-time process .ξtε,u as follows: ε,u .ξt

:= ε

2 [t/ε ]

a(zl , ul ),

ξ0ε = y,

l=0

  then under balance condition . E U π(dz)π1 (du)a(z, u) = 0 and   2 < +∞ we get that the limit process .ξ (t) = . 0 E U π(dz)π1 (du)|a(z, u)| ε limε→0 ξt has the following form: ξ0 (t) = y + bwt ,

.

where .b2 = 2aˆ 0 − aˆ 2 , and   .a ˆ0 = π(dz)π1 (du)a(z, u)R0 a(z, u), E

U

and .wt is a standard Wiener process.

  aˆ 2 =

π(dz)π1 (du)a 2 (z, u), E

U

112

6 Controlled Discrete-Time Semi-Markov Random Evolutions

Diffusion Approximation with Equilibrium of CAF Let us consider the following normalized additive functional: wtε,u := ε−1/2 [ytε,u − at]. ˆ

.

Then, this process converges to the following process, .σ wt , where   σ2 =

π(dz)π1 (du)(a(z, u) − a)R ˆ 0 (a(z, u) − a), ˆ

.

E

U

and .wt is a standard Wiener process. In this way, the AF .ytε may be presented in the following approximated form: ytε ≈ at ˆ +

.

√

εσ wt .

6.4.2 Controlled Geometric Markov Renewal Processes The CGMRP is defined in the following way (see [159, 160]): Sku := S0

k 

.

(1 + a(zl , ul )),

k ∈ N,

τ0 = s.

l=1

 We suppose that . 0k=1 = 1. If we define the operator .D(z) on .C0 (R) in the following way: D(z, u)ϕ(s) := ϕ(s(1 + a(z, u))),

.

then the discrete-time semi-Markov random evolution .Φku ϕ has the following presentation, Φku ϕ(s) = ϕ(Sku ).

.

Averaging of CGMRP Now, define the following sequence of processes: ε,u .St

:= S0

[t/ε] 

(1 + εa(zk , uk )),

t ∈ R+ ,

S0 = s.

k=1

Then, under averaging conditions, the limit process .S¯t has the following form: ˆ , S¯t = S0 eat

.

where .aˆ =

  E U

π(dz)π1 (du)a(z, u).

6.4 Applications to Stochastic Systems

113

Diffusion Approximation of CGMRP If we define the following sequence of processes:

S

.

ε,u

(t) := S0

2 [t/ε ]

(1 + εa(zk , uk )),

t ∈ R+ ,

S0 = s,

k=1

then, in the diffusion approximation scheme, we have the following limit process, S0 (t):

.

S0 (t) = S0 e−t aˆ 2 /2 eσa w(t) ,

.

where   aˆ 2 :=

π(dz)π1 (du)a 2 (z, u),

.

E

U

  2 .σa

:=

π(dz)π1 (du)[a 2 (z, u)/2 + a(z, u)R0 a(z, u)]. E

U

It means that .S0 (t) satisfies the following stochastic differential equation: .

1 dS0 (t) = (σa2 − aˆ 2 )dt + σa dwt , S0 (t) 2

where .wt is a standard Wiener process. Diffusion Approximation with Equilibrium of CGMRP Let us consider the following normalized GMRP: wtε,u := ε−1/2 [ln(Stε,u /S0 ) − at]. ˆ

.

It is worth noticing that in finance the expression .ln(Stε,u /S0 ) represents the logreturn of the underlying asset (e.g., stock) .Stε,u . Then, this process converges to the following process, .σ wt , where   σ2 =

π(dz)π1 (du)(a(z, u) − a)R ˆ 0 (a(z, u) − a), ˆ

.

E

U

and .wt is a standard Wiener process. In this way, the GMRP .Stε may be presented in the following approximated form: √ εσ wt

ˆ Stε ≈ S0 eat+

.

.

114

6 Controlled Discrete-Time Semi-Markov Random Evolutions

6.4.3 Controlled Dynamical Systems We consider here discrete-time CDS and their asymptotic behavior in series scheme: average and diffusion approximation [101]. Define the measurable function C on .R × E × U . Let us consider the difference equation ε,u yk+1 = ykε,u + εC(ykε ; zk+1 , uk+1 ),

.

k ≥ 0,

and

y0ε = u,

(6.12)

switched by the SMC .(zk ). The perturbed operators .D ε (z, u), x ∈ E, are defined now by D ε (z, u)ϕ(u) = ϕ(z + εC(z, x, u)).

.

Averaging of CDS Under averaging assumptions, the following weak convergence takes place: ε,u y[t/ε] ⇒ y(t),

.

as

ε ↓ 0,

where .y(t), t ≥ 0, is the solution of the following (deterministic) differential equation: .

where .C(z) =

  E U

d y(t) = C(y(t)), dt

and

y(0) = u,

(6.13)

π(dx)π1 (du)C(z, x, u).

Diffusion Approximation of CDS Under diffusion approximation conditions, the following weak convergence takes: place ε,u y[t/ε 2 ] ⇒ xt ,

.

as

ε ↓ 0,

where .xt , t ≥ 0, is a diffusion processes, with initial value .x0 = u, determined by the operator 1 Lϕ(z) = a(z)ϕ (z) + b2 (z)ϕ (z), 2

.

provided that .b2 (z) > 0, and drift and diffusion coefficients are defined as follows: .

b2 (z) := 2C 0 (z) − C 2 (z), a(z) := C 01 (z) − C 1 (z),

6.4 Applications to Stochastic Systems

115

with:

  C 0 (z) := E U π(dx)π1 (du)C0 (z, x, u), .C0 (z, x, u) := C(z, x, u)R0 C(z, x, u)   ∗ ∗ .C 2 (z) := E U π(dx)π1 (du)C (z, x, u)C(z, x, u), where .C means transpose of the vector C  .C 01 (z) := E U π(dx)π1 (du)C01 (z, x, u), .C01 (z, x, u) := C(z, x, u)R0 Cz .(z, x, u)   .C 1 (z) := E U π(dx)π1 (du)C1 (z, x, u), .C1 (z, x, u) := C(z, x, u)Cz (z, x, u) .

6.4.4 The Dynamic Programming Equations for Limiting Models in Diffusion Approximation In this section, we consider the DPE, i.e., HJB Equations, for the limiting models in DA from Sects. 6.4.1, 6.4.2, and 6.4.3. As long as all limiting processes in DA in Sects. 6.4.1, 6.4.2, and 6.4.3 are diffusion processes, then we will set up a general approach to control for diffusion processes, see [25]. Let .xtu be a diffusion process satisfying the following stochastic differential equation: dxtu = μ(xtu , ut )dt + σ (xtu , ut )dwt ,

.

where .ut is the control process, and .wt is a standard Wiener process. Let us also introduce the following performance criterion function, .J u (t, x):  J u (t, x) := Et,x [G(xTu ) +

T

.

t

F (s, xsu , us )ds],

where .G(x) : R → R is a terminal reward function (uniformly bounded), and F (t, x, u) : R+ × R 2 → R is a running penalty/reward function (uniformly bounded), .0 ≤ t ≤ T . The problem is to maximize this performance criterion, i.e., to find the value function

.

J (t, x) := sup J u (t, x),

.

u∈Ut,T

where .Ut,T is the admissible set of strategies/controls which are .F -predictable, nonnegative, and bounded.

116

6 Controlled Discrete-Time Semi-Markov Random Evolutions

The Dynamic Programming Principle (DPP) for diffusions states that the value function .J (t, x) satisfies the DPP  J (t, x) = sup Et,x [J u (T , xTu ) +

T

.

u∈Ut,T

t

F (s, xsu , us )ds]

for all .(t, x) ∈ [0, T ] × R. Moreover, the value function .J (t, x) above satisfies the Dynamic Programming Equation (DPE) or Hamilton–Jacobi–Bellman (HJB) equation: .

∂J (t, x) + sup [Lut J (t, x) + F (t, x, u)] = 0 ∂t u∈Ut,T

(6.14)

J (T , x) = G(x), where .Lut is an infinitesimal generator of the diffusion process .xtu above, i.e., Lut = μ(x, u)

.

6.4.4.1

σ 2 (x, u) ∂ 2 ∂ + . ∂x 2 ∂x 2

DPE/HJB Equation for the Limiting CAF in DA (see Sect. 6.4.1)

We remind that the limiting process .ξ0 (t) = limε→0 ξtε in this case has the following form: ξ0 (t) = y + bwt ,

.

where .b2 = 2aˆ 0 − aˆ 2 , and   .a ˆ0 = π(dz)π1 (du)a(z, u)R0 a(z, u), E

  aˆ 2 =

U

π(dz)π1 (du)a 2 (z, u), E

U

and .wt is a standard Wiener process. In this case, the DPE or HJB equation (6.14) reads with the generator Lut =

.

1 2 ∂2 b (u) 2 , 2 ∂x

with .b2 (u) := 2aˆ 0 (u) − aˆ 2 (u), and  .a ˆ 0 (u) := π(dz)a(z, u)R0 a(z, u), E

 aˆ 2 (u) :=

π(dz)a 2 (z, u). E

6.4 Applications to Stochastic Systems

6.4.4.2

117

DPE/HJB Equation for the Limiting CGMRP in DA (see Sect. 6.4.2)

We recall that we have the following limiting process: .S0 (t) in this case: S0 (t) = S0 e−t aˆ 2 /2 eσa w(t) ,

.

where   aˆ 2 :=

π(dz)π1 (du)a 2 (z, u),

.

E

U

  σa2 :=

π(dz)π1 (du)[a 2 (z, u)/2 + a(z, u)R0 a(z, u)].

.

E

U

Furthermore, .S0 (t) satisfies the following stochastic differential equation (SDE): .

1 dS0 (t) = (σa2 − aˆ 2 )dt + σa dwt , S0 (t) 2

where .wt is a standard Wiener process. In this case, the DPE or HJB equation (6.14) reads with the generator 1 ∂2 ∂ 1 2 (σa (u) − a2 (u)) + σa2 (u) 2 , ∂s 2 2 ∂s   and .aˆ 2 (u) := E π(dz)a 2 (z, u), .σa2 (u) := E π(dz)[a 2 (z, u)/2+a(z, u)R0 a(z, u)]. Lut =

.

6.4.4.3

DPE/HJB Equation for the Limiting CDS in DA (see Sect. 6.4.3)

We remind that in the diffusion approximation the limiting process is a diffusion process .xt with a generator 1 Lϕ(z) = a(z)ϕ (z) + b2 (z)ϕ (z), 2

.

provided that .b2 (z) > 0, and drift and diffusion coefficients are defined as follows: .

b2 (z) := 2C 0 (z) − C 2 (z), a(z) := C 01 (z) − C 1 (z),

118

6 Controlled Discrete-Time Semi-Markov Random Evolutions

with:

  C 0 (z) := E U π(dx)π1 (du)C0 (z, x, u), .C0 (z, x, u) := C(z, x, u)R0 C(z, x, u)   ∗ ∗ .C 2 (z) := E U π(dx)π1 (du)C (z, x, u)C(z, x, u), where .C means transpose of the vector C  .C 01 (z) := E U π(dx)π1 (du)C01 (z, x, u), .C01 (z, x, u) := C(z, x, u)R0 Cz .(z, x, u)   .C 1 (z) := E U π(dx)π1 (du)C1 (z, x, u), .C1 (z, x, u) := C(z, x, u)Cz (z, x, u) .

In this case the DPE or HJB equation (6.14) reads with the generator 1 Lut = a(z, u)ϕ (z) + b2 (z, u)ϕ (z), 2

.

and .

b2 (z, u) := 2C 0 (z, u) − C 2 (z, u), a(z, u) := C 01 (z, u) − C 1 (z, u),

with:

 C 0 (z, u) := E π(dx)C0 (z, x, u), .C0 (z, x, u) := C(z, x, u)R0 C(z, x, u)  ∗ ∗ .C 2 (z, u) := E π(dx)C (z, x, u)C(z, x, u), where .C means transpose of the vector C  .C 01 (z, u) := π(dx)C01 (z, x, u), .C01 (z, x, u) := C(z, x, u)R0 Cz (z, x, u) E .C 1 (z, u) := E π(dx)C1 (z, x, u), .C1 (z, x, u) := C(z, x, u)Cz (z, x, u) .

Remark 6.1 Our construction here is equivalent to some extent to “Recurrent Processes of a semi-Markov (RPSM)” type studied first in [6, 7] including limit theorems. Those results were described in more detail in [4, 5]. In particular, “RPSM with Markov switching” reflects the case of independent Markov components .zk and .uk , and “General case of RPSM” reflects the case when .uk is dependent on .zk .

6.5 Solution of Merton Problem for the Limiting CGMRP in DA In this section we present some basics associated with the Merton portfolio optimization problem (MPOP) and show how to solve it for the limiting CGMRP in DA. It includes short introduction to the problem, utility function’s definition and examples, wealth process’ definition and its equation, solution of MPOP for those utility functions, and, finally, solution of MPOP for the limiting CGMRP in DA scheme. For more details on general theory of MPOP, see [79, 112–114].

6.5 Solution of Merton Problem for the Limiting CGMRP in DA

119

6.5.1 Introduction We consider a continuous-time .(B, S) security/market model with risk-less asset priced by .Bt (e.g., bond, bank account) and risky asset priced by .St (e.g., a stock), where the investor can re-balance his wealth/capital/value of portfolio .Xt at any time .t, before the terminal time .T , by moving capital from risk-less security to risky and vice versa without any transaction costs. We note that the change of .Xt can be expressed in the following way: dXt = πtB dBt + πtS dSt ,

.

X0 = x,

where .πtB and .πtS denote the number of the bonds and stocks, respectively, held by the investor at time .t. We note that both .πtB and .πtS are nonnegative, but not necessarily integer. We also suppose that the wealth .Xt is always nonnegative: .Xt ≥ 0, 0 ≤ t ≤ T . Merton portfolio optimization problem (MPOP) (see [112, 113]) aims to find the optimal investment strategy for the investor with those two objects of investment, namely risk-less asset (e.g., a bond or bank account), paying a fixed rate of interest .r, and a number of risky assets (e.g., stocks) whose price follows a geometric Brownian motion. In this way, we suppose that .Bt and .St satisfy the following ODE and SDE (i.e., geometric Brownian motion (GBM)), respectively:

.

dBt = rBt dt dSt = St (μdt + σ dWt ),

(6.15)

where .Wt is a standard Wiener process, .r > 0 is the interest rate, .μ ∈ R is an appreciation rate, and .σ > 0 is a volatility. The investor starts with an initial amount of money, say .X0 = x, and wishes to decide how much money to invest in risky and risk-less assets to maximize the final wealth .Xt at the maturity .T . We suppose that the investor is risk averse meaning that he/she refrains from investing in assets which have a high risk of losing money (even with high return). It also means that he/she will only accept investments which are better than fair game. How to measure the investor attitude to risk and individual preferences? The answer lies in so-called utility function: using this function the expected return of the investment is not maximized but rather the utility function, making it possible not only maximizing the expected value of the final wealth but also limiting the risk of losing money simultaneously.

6.5.2 Utility Function The concept of utility function is needed to characterize the investor’s decisions and preferences. We use the following notation .U (x) for the utility function. This

120

6 Controlled Discrete-Time Semi-Markov Random Evolutions

function expresses how satisfied the investor is with a possible outcome of the investment. Thus, this is usually a function of the wealth .Xt or a function of the consumption .ct . If the investor is risk averse (see the Introduction section), then it means that the utility function should be strictly concave. If we take into account that the investor always prefers to have more wealth than less (we can refer to as non-satisfaction of the investor), then it implies that the utility function should be strictly monotone increasing. Thus, we arrive to the following definition of a utility function. Definition 6.2 (Utility Function) A utility function is defined as a function .U (x) : S → R+ which is strictly increasing, strictly concave, and continuous on .S ⊂ R+ . There are several coefficients associated with the utility functions, and among of them are ARAC and RRAC. Definition 6.3 (Absolute Risk Aversion Coefficient (ARAC)) For a utility function .U (x), the ARAC is defined as the ratio: ARAC(x) := −

.

U (x) . U (x)

Definition 6.4 (Relative Risk Aversion Coefficient (RRAC)) For the utility function .U (x), the RRAC is defined as the ratio: RRAC(x) := −

.

xU (x) . U (x)

There are many examples of utility functions. Among them we would like to mention: - Logarithmic utility function: .U (x) = log(x) p - Power utility function: .U (x) = xp , p < 1, p ∈ R, p = 0 - Exponential utility function: .−e−px , p > 0 We will show below how to solve MPOP for these utility functions. We note that the first two classes of utility functions above define so-called hyperbolic absolute risk aversion (HARA) and constant relative risk aversion (CRRA) classes, respectively.

6.5.3 Value Function or Performance Criterion Let us consider Merton portfolio optimization problem. As for the control at time t we will take the function .πt , the fraction of the total wealth which should be invested

6.5 Solution of Merton Problem for the Limiting CGMRP in DA

121

in risky assets. The, the number of the bonds and stocks, respectively, will be .

(1 − πt )Xt Bt

and

πt Xt . St

Then the change of the wealth process .Xt can be rewritten in the following way (taking into account (6.15) system above): dXt = Xt [(r + πt (μ − r))dt + πt σ dWt ].

.

To stress dependency of .Xt on .πt , we will use the notation .Xtπ instead of .Xt . Thus, π .Xt satisfies dXtπ = Xtπ [(r + πt (μ − r))dt + πt σ dWt ].

.

(6.16)

Our main goal is to solve the following optimization problem: .

max E[U (XTπ )|X0 = x], π

meaning to maximize the wealth/value function or performance criterion E[U (XTπ )|X0 = x]. Depending on different utility functions .U (x) we have different solution to the optimization problem.

.

6.5.4 Solution of Merton Problem: Examples As we mentioned above, for different utility functions .U (x), we have different solutions to the optimization problem. Logarithmic Utility Function .U (x) = log(x). If we take the logarithmic utility function .U (x) = log(x), then the solution for the Merton problem will be πt∗ =

.

μ−r . σ2

Solution As long as we need to maximize .E[log(XTπ )|X0π = x], the idea of finding t the maximum is the following one: (1) solve (6.16) for .Xtπ = x exp{ 0 (r + (μ − t r)πs − 12 σ 2 πs2 )ds + 0 σ πs dWs } (we used Itô formula), (2) maximize .r + (μ − r)πs − 12 σ 2 πs2 (last term in the .exp function is a martingale, and thus we need only to maximize the first term), and (3) the maximum of this is achieved at .πt∗ = (μ − r)/σ 2 (take the first derivative by .π and equate to zero).

122

6 Controlled Discrete-Time Semi-Markov Random Evolutions

Power Utility Function .U (x) = x p /p. If we take the power utility function .U (x) = x p /p, then the solution to the Merton problem will be πt∗ =

.

μ−r σ 2 (1 − p)

.

Solution The solution here is given in the same way as for the logarithmic utility function with some obvious modifications regarding steps (1)–(3) (e.g., take into account that Itô formula for power utility function is different). As we could see, we get the same result that the optimal strategy is a constant fraction of the wealth in stocks. See [79] for more details. In the next section, we consider the solution of Merton problem for the case of limiting CGMRP in DA with exponential utility function to show the wide variety of utility function usage. Remark 6.2 (Financial Market with One Bond and n Stocks) Let us consider n stocks with prices .St = (St1 , . . . , Stn ) that follow the dynamics dSti = Sti (μi dt +

n 

.

σik dWtk ),

S0i = s0i ,

i = 1, 2, . . . , n,

k=1

or, in matrix form, dSt = Diag(St )(μdt + ΣdWt ),

.

where .μ ∈ Rn , .Σ is a non-singular volatility matrix in .Rn×n , and .Wt is a vector of n independent standard Wiener processes .Wti . The bond price .Bt follows the following dynamics: dBt = rBt dt,

.

or

Bt = B0 ert .

If we take for the control at time t an n-dimensional process .π t = (πt1 , . . . , πtn ), where .πti denotes the fraction of wealth which is invested in stock .i, then the optimal solution for the logarithmic utility function is πt∗ = (ΣΣ T )−1 (μ − r),

.

t ∈ [0, T ],

where .Σ T means transpose matrix to .Σ, and .r = (r, . . . , r). Similarly, the optimal solution for the power utility function is πt∗ = (ΣΣ T )−1 (μ − r)/(1 − p),

.

t ∈ [0, T ], p < 1, p = 0.

6.5 Solution of Merton Problem for the Limiting CGMRP in DA

123

6.5.5 Solution of Merton Problem Let us consider the Merton portfolio optimization problem applied to the limiting CGMRP in DA above, see Sect. 5.2. For the solution of MPOP, in this case we use exponential utility function to make a difference between this case and other two above for logarithmic and power utility functions. We recall, in this problem, the agent seeks to maximize expected wealth by trading in a risky asset and the risk-free bonds (e.g., bond or bank account). She/he places portion .πt of a total wealth .Xtπ in the risky asset .S0 (t) and looks to obtain the value function (performance criterion) J π (t, S, x) := sup Et,S,x [U (XTπ )],

.

π ∈U0,T

which depends on the current wealth .X0π = x and asset price .S0 (0) = S, and the optimal trading strategy .π, .U (x) is the agent’s utility function (e.g., exponential −γ x ) or power .x p /p, or logarithmic .log(x), etc.). We suppose that the asset .(−e price .S0 (t) satisfies the following SDE (or GBM): .

dS0 (t) = (μ − r)dt + σa dWt , S0 (t)

S0 (0) = S,

where μ :=

.

1 2 (σ − aˆ 2 ), 2 a

  aˆ 2 :=

π(dz)π1 (du)a 2 (z, u),

.

E

U

  2 .σa

:=

π(dz)π1 (du)[a 2 (z, u)/2 + a(z, u)R0 a(z, u)]. E

U

Here, .μ represents the expected continuously compounded rate of growth of the traded asset, and r is the continuously compounded rate of return of the risk-free asset (bond or bank account). The wealth process .Xtπ follows the following SDE: dXtπ = (πt (μ − r) + rXtπ )dt + πt σa dWt ,

.

X0π = x.

From the SDEs for .S0 (t) and for .Xtπ above, we conclude that the infinitesimal generator for the pair .(S0 (t), Xtπ ) is Lπt = (rx + (μ − r)π )

.

1 1 ∂2 ∂ ∂2 ∂ ∂2 + σa2 π 2 + (μ − r)S + σa2 S 2 2 + σa π . ∂x 2 ∂S 2 ∂x∂S ∂x ∂S

124

6 Controlled Discrete-Time Semi-Markov Random Evolutions

From HJB equation for the limiting CGRMP in DA, it follows that the value function J (t, S, x) = sup J π (t, S, x)

.

π ∈Ut,T

should satisfy the equation .

∂J (t, S, x) + sup[Lπt J (t, S, x)] = 0 ∂t π

with terminal condition .J (T , S, x) = U (x). The explicit solution of this PDE depends on the explicit form of the utility function .U (x). To make a difference between previous cases with logarithmic and power utility functions, let us take the exponential utility function U (x) = −e−γ x ,

.

γ > 0,

x ∈ R.

In this case we can find that the optimal amount to invest in the risky asset is a deterministic function of time πt∗ =

.

(1/2)(σa2 − aˆ 2 ) − r −r(T −t) e . γ σa2

The solution of this problem follows similar steps as in 1. above for the logarithmic utility function (see section 0.4 Solution of Merton Problem: Examples) with obvious modifications.

6.6 Rates of Convergence in Averaging and Diffusion Approximations The rate of convergence in a limit theorem is important in several ways, both theoretical and practical. We present here the rates of convergence of CDTSMRE in the averaging, diffusion approximation, and diffusion approximation with equilibrium schemes and, as corollaries, we give the rates of convergence for CAF and CGMRP in the corresponding limits. Proposition 6.1 The Rate of Convergence of CDTSMRE in the Averaging has the following form: ε,u ||E[Φ[t/ε] ϕ] − Φ(t)ϕ|| ≤ εA(T , ϕ, ||R0 ||, ||D1 ||),

.

where .A(T , ϕ, ||R0 ||, ||D1 ||) is a constant, and .0 ≤ t ≤ T . The proof of this proposition is given in Sect. 6.7.4.

6.6 Rates of Convergence in Averaging and Diffusion Approximations

125

Proposition 6.2 The Rate of Convergence of CDTSMRE in the Diffusion Approximation takes the following form: ε,u ||E[Φ[t/ε 2 ] ϕ] − Φ0 (t)ϕ|| ≤ εD(T , ||ϕ||, ||R0 ||, ||D1 ||, ||D2 ||),

.

where .D(T , ||ϕ||, ||R0 ||, ||D1 ||, ||D2 ||) is a constant, and .0 ≤ t ≤ T . Proposition 6.3 The Rate of Convergence of CDTSMRE in Diffusion Approximation with Equilibrium has the following form: ||E[Wtε,u ϕ] − Wt0 ϕ|| ≤

√

.

εN(T , ||ϕ||, ||R0 ||, ||D1 ||, ||D12 ||),

where .N (T , ||ϕ||, ||R0 ||, ||D1 ||, ||D12 ||) is a constant and .0 ≤ t ≤ T . The proofs of the above Propositions 6.2 and 6.3 are similar to the proof of Proposition 6.1. We give in what follows some rate of convergence results (Corollaries 6.1 and 6.2) concerning applications. Corollary 6.1 The Rate of Convergence in the Limit Theorems for CAF: - Rate of Convergence in Averaging ||Eytε,u − y0 (t)|| ≤ εa(T , ||R0 ||, ||a||),

.

where .a(T , ||R0 ||, ||a||) is a constant, and .0 ≤ t ≤ T . - Rate of Convergence in Diffusion Approximation ||Eξtε,u − ξ0 (t)|| ≤ εd(T , ||R0 ||, ||a||, ||a 2 ||),

.

where .d(T , ||R0 ||, ||a||, ||a 2 ||) is a constant, and .0 ≤ t ≤ T . - Rate of Convergence in diffusion approximation with equilibrium for CAF ||EWtε,u − wt || ≤

.

√

εn(T , ||R0 ||, ||a||, ||a 2 ||),

where .n(T , ||R0 ||, ||a||, ||a 2 ||) is a constant, and .0 ≤ t ≤ T . Corollary 6.2 The Rate of Convergence in the Limit Theorems for CGMRP: - Rate of Convergence in Averaging ||EStε,u − τ¯t || ≤ εa(T , ||R0 ||, ||a||),

.

where .a(T , ||R0 ||, ||a||) is a constant, and .0 ≤ t ≤ T . - Rate of Convergence in Diffusion Approximation ||EStε,u − τ0 (t))|| ≤ εd(T , ||R0 ||, ||a||, ||a 2 ||),

.

where .d(T , ||R0 ||, ||a||, ||a 2 ||) is a constant, and .0 ≤ t ≤ T .

126

6 Controlled Discrete-Time Semi-Markov Random Evolutions

- Rate of Convergence in diffusion approximation with equilibrium ||EWtε,u − wt || ≤

.

√

εn(T , ||R0 ||, ||a||, ||a 2 ||),

where .n(T , ||R0 ||, ||a||, ||a 2 ||) is a constant, and .0 ≤ t ≤ T .

6.7 Proofs The proofs here have almost the same general construction scheme as in our paper [104] except that we consider also the control process. Let .CB [0, ∞) be the space of B-valued continuous functions defined on .[0, ∞).

6.7.1 Proof of Theorem 6.1 The proof of the relative compactness of CDTSMRE in the average approximation is based on the following four lemmas. ε,u The CDTSMRE .Φ[t/ε] ϕ, see (6.1), is weakly compact in .DB [0, ∞) with limit points into .CB [0, ∞). ε,u Lemma 6.1 Under Assumptions A1–A7, the limit points of .Φ[t/ε] ϕ, .ϕ ∈ B0 , as .ε → 0, belong to .CB [0, ∞).

Proof Assumptions A5–A6 imply that the discrete-time semi-Markov random evolution .Φku ϕ is a contractive operator in H and, therefore, .||Φku ϕ||H is a supermartingale for any .ϕ ∈ H, where .||·||H is a norm in Hilbert space H ([91, 101]) ε,u Obviously, the same properties satisfy the following family .Φ[t/ε] . Using Doob’s ε,u inequality for the supermartingale .||Φ[t/ε] ||H , we obtain ε,u P{Φ[t/ε] ∈ KΔ } ≥ 1 − Δ,

.

where .KΔ is a compact set in .B and .Δ is any small number. It means that sequence ε,u Φ[t/ε] is tight in .B. Taking into account conditions A1–A6, we obtain that discreteε,u time semi-Markov random evolution .Φ[t/ε] is weakly compact in .DB [0, +∞) with limit points in .CB[0, +∞), .ϕ ∈ B0 .    ε,u ε,u ε,u  Let .Jtε,u := J (Φ[t/ε] ; [t/ε]) := supk≤[t/ε] Φ[t/ε]+k ϕ − Φ[t/ε] ϕ , and let .KΔ be a compact set from compact containment condition .Δ > 0. It is sufficient to show that .Jtε,u weakly converges to zero. This is equivalent to the convergence of .Jtε,u in probability as .ε → 0.

.

6.7 Proofs

127

From the very definition of .Jtε,u and A3, we obtain   Jtε,u 1KΔ ≤ ε sup sup (D1 (zk , uk )ϕ + D0ε (zk , uk )ϕ ),

.

k≤[t/ε] ϕ∈SΔ

where .1KΔ is the indicator of the set .KΔ , and .SΔ is the finite .δ-set for .KΔ . Then, for .δ < Δ, we have Pπ ×π1 (Jtε,u 1KΔ > Δ) ≤ Pπ ×π1 ( sup Dk > (Δ − δ)/ε)

.

k≤[t/ε]

=

[t/ε] 

Pπ ×π1 ({ sup Dk > (Δ − δ)/ε} ∩ Di ) k≤[t/ε]

i=1

[t/ε] (D1 (x, u)ϕ2 ≤ ε2 [t/ε] sup [P ϕ∈SΔ

  2  +2 D1 (x, u)ϕ D0ε (x, u)ϕ  + D0ε (x, u)ϕ  )],   where .Dk := supϕ∈SΔ (D1 (zk , uk )ϕ + D0ε (zk , uk )ϕ ), and Di := {ω : Dk contains the maximum for the first time on the variable Di }.

.

k is bounded when .k → ∞. So is the case It is worth noticing that the operator .P [t/ε]  for .P when .ε → 0. Taking both .ε and .δ go to 0, we obtain the proof of this lemma. Let us now consider the continuous-time martingale ε,u .Mt

:=

ε M[t/ε]

=

ε,u Φ[t/ε]

−I −

[t/ε]−1 

ε,u Eπ ×π1 [Φk+1 − Φkε,u | Fk ].

(6.17)

k=0

Lemma 6.2 The process ε,u Mtε,u := Φ[t/ε] −I −

[t/ε]−1 

.

D ε (·, u) − I ]Φ ε,u [P

=0 u -martingale. is an .F[t/ε]

 ε,u  ε Proof As long as .Mkε,u := Φkε,u − I − k−1 =0 [P D (·, u) − I ]Φ is a martingale, ε,u ε,u u ε .Mt = M[t/ε] is an .F[t/ε] -martingale. Here, we have .Eπ ×π1 [Mk+1 | Fku ] = Mkε,u , which can be easily checked. [t/ε] ε,u Lemma 6.3 The family . ( k=0 Eπ ×π1 [Φk+1 ϕ − Φkε,u ϕ | Fku ]) is relatively ∗ compact for all . ∈ B0 , dual of the space .B0 .

128

6 Controlled Discrete-Time Semi-Markov Random Evolutions

Proof Let Ntε,u :=

[t/ε] 

.

ε,u Eπ ×π1 [(Φk+1 − Φkε,u )ϕ | Fku ].

k=0

Then, Ntε,u =

[t/ε] 

.

D ε (·, u) − I ]Φ ε,u . [P k

k=0 ε,u As long as .Φk+1 = D ε (zk+1 , uk+1 )Φkε,u , we obtain ε,u Eπ ×π1 [Φk+1 ϕ | Fku ] = Eπ ×π1 [D ε (zk+1 , uk+1 )Φkε,u ϕ | Fku ].

.

Then, [(t+η)/ε]  ε,u ε,u u . Eπ ×π1 [Φk+1 ϕ − Φk ϕ | Fk ]) ( k=[t/ε]+1 [(t+η)/ε]  D ε (zk+1 , zk+1 ) − I ]Φ ε,u ϕ) = ( [P k k=[t/ε]+1   (D1 (zk+1 , uk+1 ) + D ε (zk+1 , uk+1 ))ϕ  ≤ ε   ([(t + η)/ε] − [t/ε] − 1) P 0  η  ≤ ε   P (D1 (·, u) + D0ε (·, u))ϕ  ε   (D1 (·, u) + D ε (·, u))ϕ  → 0, η → 0, = η   P 0   (D1 (·, u) + D ε (·, u))ϕ  is bounded for any .ϕ ∈ B0 . as .P 0 [t/ε] ε,u It means that the family . ( k=0 Eπ ×π1 [Φk+1 ϕ − Φkε,u ϕ | Fku ]) is relatively ∗ compact for any . ∈ B0 . ε,u Lemma 6.4 The family . (M[t/ε] ϕ) is relatively compact for any . ∈ B∗0 and any .ϕ ∈ B0 . ε,u Proof It is worth noticing that the martingale .M[t/ε] can be represented in the form of the martingale differences

ε,u M[t/ε] =

[t/ε]−1 

.

k=0

ε,u ε,u Eπ ×π1 [Φk+1 ϕ − Eπ ×π1 (Φk+1 ϕ | Fku )].

6.7 Proofs

129

Then, using the equality ε,u Eπ ×π1 [Φk+1 ϕ | Fku ] = Eπ ×π1 [D ε (zk+1 , uk+1 )Φkε,u ϕ | Fku ],

.

we get ε,u ε,u M[(t+η)/ε] ϕ − M[t/ε] ϕ=

[(t+η)/ε] 

.

[D ε (zk+1 , uk+1 )Φkε,u ϕ

k=[t/ε]+1

−Eπ ×π1 [D ε (zk+1 , uk+1 )Φkε,u ϕ | Fku ]} =

[(t+η)/ε] 

[D ε (zk+1 , uk+1 )Φkε,u ϕ

k=[t/ε]+1

D ε (zk+1 , uk+1 )Φ ε,u ϕ] −P k =

[(t+η)/ε] 

D ε (zk+1 , uk+1 )]Φ ε,u ϕ, [D ε (zk+1 , uk+1 ) − P k

k=[t/ε]+1

for any .η > 0. Now, from the above, we get .

ε,u ε,u Eπ ×π1 (M[(t+η)/ε] ϕ − M[t/ε] ϕ)

  ≤ ([t + η)/ε] − [t/ε])εEπ ×π1 (D1 (zk+1 , uk+1 )ϕ + D0ε (zk+1 , uk+1 )ϕ      D ε (·, u)ϕ ) D1 (·, u)ϕ  + P + P 0     D ε (·, u)ϕ ) → 0, η → 0, D1 (·, u)ϕ  + P ≤ 2η(P 0 which proves the lemma. Now the proof of Theorem 6.1 is achieved as follows. From Lemmas 6.2, 6.3, and 6.4 and the representation (6.17), it follows that the ε,u family . (Φ[t/ε] ϕ) is relatively compact for any . ∈ B∗0 and any .ϕ ∈ B0 . Moreover, let .Lε (x), .x ∈ E, be a family of perturbed operators defined on .B as follows: D ε (x, u). +P D1 (x, u) + P Lε,u (x) := ε−1 Q 0

.

(6.18)

Then, the process ε,u Mtε,u = Φ[t/ε] −I −ε

[t/ε]−1 

.

=0

is an .Ftε,u -martingale.

Lε,u Φ ε,u

(6.19)

130

6 Controlled Discrete-Time Semi-Markov Random Evolutions

The following singular perturbation problem, for the non-negligible part of −1   compensating operator, .Lε,u , denoted by .Lε,u 0 (x) := ε Q + P D1 (x, u): ε ε,u Lε,u , 0 ϕ = Lϕ + εθ

(6.20)

.

on the test functions .ϕ ε (z, x) = ϕ(z) + εϕ1 (z, x), has the solution (see [90] Proposition 5.1):  ϕ ∈ N (Q),

.

ϕ1 = R0 D˜ 1 ϕ,

with D1 (x, u) − D 1 , D˜ 1 (x, u) = P

.

1 = D

 π × π1 (dx)D1 (x, u) E

and θ ε,u (x) = (P  × P u )D1 (x, u)R0 D˜ 1 (x, u)ϕ.

.

The limit operator is then given by LΠ = Π D1 (·, u)Π,

.

(6.21)

form which we get the contracted limit operator 1 . L=D

(6.22)

.

We note that martingale .Mtε,u has the following asymptotic representation: ε,u Mtε,u = Φ[t/ε] −I −ε

[t/ε]−1 

.

LΦ ε,u + Oϕ (ε),

(6.23)

=0

[t/ε]−1 where .||Oϕ (ε)|| → 0, as .ε → 0. The families .l(M[t/ε] ) and .l( =0 [(P  × ε,u P u )D ε (·, u) − I ]Φ ) are weakly compact for all .l ∈ B∗0 in a dense subset ∗ ⊂ B. It means that family .l(Φ ε,u ) is also weakly compact. In this way, the .B [t/ε] 0 t [t/ε]−1 LΦ ε,u ϕ converges, as .ε → 0, to the integral . 0 LΦ(s)ϕds. The sum .ε =0 quadratic variation of the martingale .l(Mtε,u ϕ) tends to zero when .ε → 0, thus, ε,u .Mt ϕ → 0 when .ε → 0, for any .f ∈ B0 and for any .l ∈ B∗0 . Passing to the limit in ε,u ϕ →ε→0 Φ(t)ϕ, where .Φ(t) is defined in (6.4). (6.23), when .ε → 0, we get .Φ[t/ε] The quadratic variation of the martingale .Mtε,u , in the average approximation, is ε,u  (M[t/ε] =

[t/ε] 

.

k=0

ε,u ε Eπ ×π1 [ 2 (Mk+1 ϕ − Mkε,u ϕ ε ) | Fku ],

(6.24)

6.7 Proofs

131

where .ϕ ε (x) = ϕ(x) + εϕ1 (x). Hence ε,u ε ε,u ε,u (Mk+1 ϕ − Mkε,u ϕ ε ) = ((Mk+1 − Mkε,u )ϕ) + ε ((Mk+1 − Mkε,u )ϕ1 ),

.

and ε,u ε,u ε,u Mk+1 − Mkε,u = Φk+1 − Φkε,u − Eπ ×π1 [Φk+1 − Φkε,u | Fku ].

.

(6.25)

Therefore, ε,u ε (Mk+1 ϕ − Mkε,u ϕ ε ) = ((D(zk+1 , uk+1 )ε − I )Φkε,u ϕ)

.

−Eπ ×π1 [(D ε (zk+1 , uk+1 ) − I )ϕ | Fku ] +ε ((D ε (zk+1 , uk+1 ) − I )ϕ1 ) −Eπ ×π1 [(D(zk+1 , uk+1 )ε − I )ϕ1 | Fku ] = ε ((D1 (zk+1 , uk+1 ) + D0ε (zk+1 , uk+1 ))ϕ) −εEπ ×π1 [(D1 (zk+1 , uk+1 ) + D0ε (zk+1 , uk+1 ))ϕ

(6.26) | Fku ]

+ε2 ((D1 (zk+1 , uk+1 ) + D0ε (zk+1 , uk+1 ))ϕ1 ) −ε2 Eπ [(D1 (zk+1 , uk+1 ) + D0ε (zk+1 , uk+1 ))ϕ1 | Fku ]. Now, from (6.24) and (6.26) and from boundedness of all operators in (6.26) with ε,u respect to .Eπ ×π1 , it follows that . (M[t/ε]  goes to 0 when .ε → 0, and the quadratic

variation of limit process .Mt0,u , for the martingale .Mtε,u , is equal to 0. In this case, the limit martingale .Mt0 equals 0. Therefore, the limit equation for ε,u .Mt has the form (6.4). As long as the solution of the martingale problem for L is unique, then it follows that the solution of Equation (6.4) is unique as operator . ˆ is a first-order operator (.D 1 , see well [45, 153]. It is worth noticing that operator .L (6.22)). Finally, the operator .L generates a semigroup, then .Φ(t)ϕ = exp[Lt]ϕ, and the latter representation is unique.

6.7.2 Proof of Theorem 6.2 ε,u We can prove the relative compactness of the family .Φ[t/ε 2 ] exactly on the same way and following the same steps as above. However, in the case of diffusion approximation, the limit continuous martingale .M0 (t) for the martingale .Mtε has

132

6 Controlled Discrete-Time Semi-Markov Random Evolutions

quadratic variation that is not zero, that is,  M0 (t)ϕ = Φ0 (t)ϕ − ϕ −

t

.

LΦ0 (s)ds

0

and so . (M0 ) = 0, for . ∈ B∗0 . L defined in Theorem 6.2 is a second-order kind operator as Moreover, operator . 2 and .Π D1 R0 P D1 Π , compare with the first-order operator . L it contains operator .D in (6.22). Let .Lε,u (x), .x ∈ E, be a family of perturbed operators defined on .B as follows: D2 (x, u) + P D ε (x, u).  + ε−1 P D1 (x, u) + P Lε,u (x) := ε−2 Q 0

.

(6.27)

Then, the process ε,u .Mt

ε,u Φ[t/ε ε]

=

−I −ε

2

2 ]−1 [t/ε 

Lε,u Φkε,u

(6.28)

k=0

is an .Ftε,u -martingale with mean value zero. For the non-negligible part of compensating operator, .Lε,u , denoted by ε,u −2  −1   .L 0 (x) := ε Q + ε P D1 (x, u) + P D2 (x, u), consider the following singular perturbation problem ε ε,u Lε,u (x), 0 ϕ = Lϕ + εθ

.

(6.29)

where .ϕ ε (z, x) = ϕ(z) + εϕ1 (z, x) + ε2 ϕ2 (z, x). The solution of this problem is realized by the vectors (see [90], Proposition 5.2) D1 (x, u)ϕ, ϕ1 = R0 P

.

˜ ϕ2 = R0 Aϕ,

˜ ˆ Finally, the negligible term .θ ε,u (x) is with .A(x, u) := A(x, u) − A. D1 (x, u) + εP D2 (x, u)]ϕ2 + P D2 (x, u)ϕ1 . θ ε,u (x) = [P

.

 Of course, .ϕ ∈ N (Q). Now the limit operator .L is given by D2 (·, u) + P D1 (·, u)R0 P D1 (·, u), L=P

.

(6.30)

 is from which, the contracted operator on the null space .N (Q) 2 Π + Π D1 (x, u)R0 P D1 (x, u)Π. L=D

.

Moreover, due to the balance condition (6.6), we get the limit operator.

(6.31)

6.7 Proofs

133

We worth noticing that Assumptions A5–A7 and D1–D3 imply that discreteε,u time semi-Markov random evolution .Φ[t/ε 2 ] ϕ is a contractive operator in H and,

ε,u therefore, .||Φ[t/ε 2 ] ϕ||H is a supermartingale for any .ϕ ∈ H, where .|| · ||H is a norm in Hilbert space H ([91, 101]). By Doob’s inequality for the supermartingale ε,u .||Φ || , we obtain [t/ε2 ] H ε,u 1 P{Φ[t/ε 2 ] ∈ KΔ } ≥ 1 − Δ,

.

1 is a compact set in .B and .Δ is any positive small real number. where .KΔ We conclude that under Assumptions A5–A7 and D1–D3, the family .Mtε,u is tight and is weakly compact in .DB [0, +∞) with limit points in .CB [0, +∞). Moreover, under Assumptions A5–A6 and D1–D2, the martingale .Mtε,u has the following asymptotic presentation:

ε,u .Mt ϕ

=

ε,u Φ[t/ε 2]ϕ

−ϕ−ε

2

2 ]−1 [t/ε 

LΦkε,u ϕ + Oϕ (ε),

(6.32)

k=0

[t/εε ]−1 LΦkε,u ϕ) where .||Oϕ (ε)|| → 0, as .ε → 0. The families .l(Mtε,u φ) and .l(ε2 k=0 ε,u ∗ are weakly compact for all .l ∈ B and .ϕ ∈ B0 . It means that .Φ[t/ε2 ] is also weakly compact and has a limit. [t/εε ]−1 LΦkε,u ϕ Let us denote the previous limit by .Φ0 (t), then the sum .ε2 k=0 t LΦ0 (s)ϕds. Let .M0 (t) also be a limit martingale for converges to the integral . 0 ε,u .Mt when .ε → 0. Then, from the previous steps and (6.32), we obtain 

t

M0 (t)ϕ = Φ0 (t)ϕ − ϕ −

.

LΦ0 (s)ϕds.

(6.33)

0

As long as martingale .Mtε,u has mean value zero, the martingale .M0 (t) has also mean value zero. If we take the mean value from both parts of (6.33), we get 

t

0 = EΦ0 (t)ϕ − ϕ −

.

LEΦ0 (t)ϕds,

(6.34)

0

or, solving it, we get EΦ0 (t)ϕ = exp[ Lt]ϕ.

.

(6.35)

L generates a semigroup, namely, The last equality means that the operator . U (t) := EΦ0 (t)ϕ = exp[ Lt]ϕ. Now, the uniqueness of the limit evolution .Φ0 (t) in diffusion approximation follows from the uniqueness of solution of the martingale problem for .Φ0 (t) (uniqueness of the limit process under weak compactness). As

.

134

6 Controlled Discrete-Time Semi-Markov Random Evolutions

L is unique, then it long as the solution of the martingale problem for operator . follows that the solution of Equation (6.34) is unique as well [45, 153].

6.7.3 Proof of Theorem 6.3 We note that .Wtε,u in (6.10) has the following presentation: Wtε,u = ε−1/2 {



[t/ε] 

[D ε (zk−1 , uk−1 ) − I ]Φkε,u −

.

k=1

t

D 1 Φ(s)ds}.

(6.36)

0

1 ) = 0 holds, then we apply the diffusion As the balance condition .Π (D1 − D approximation algorithm (see Sect. 6.3.2), i.e., to the right-hand side of (6.36) with the following operators, .D2 = 0 and .(D1 (z) − D 1 ) instead of .D1 (z). It is worth mentioning that the family .Wtε,u is weakly compact and the result is proved (see Sect. 6.7.1 and 6.7.2).

6.7.4 Proof of Proposition 6.1 The proof of this proposition is based on the estimation of ε,u ε ||Eπ [Φ[t/ε] ϕ ] − Φ(t)ϕ||,

.

for any .ϕ ∈ B0 , where .ϕ ε (x) = ϕ(x) + εϕ1 (x). We note that  − I )ϕ1 (x) = −(Dˆ 1 − P D1 (x, u))ϕ. (P

.

(6.37)

As long as .Π (Dˆ 1 − (P  × P u )D1 (x, u))ϕ = 0, .ϕ ∈ B0 , equation (6.37) has the  − I ), .ϕ1 (x) = R0 D˜ 1 ϕ. solution in domain .R(P In this way,   Eπ ×π1 ϕ1 (x) ≤ 2 R0 

.

E

  D1 (·, u)ϕ  := 2C1 (ϕ1 R0 ), π(dz)π1 (du) P

U

(6.38)  := P − I. where .R0 is a potential operator of .Q

6.7 Proofs

135

From here, we obtain    ε,u  Eπ ×π1 (Φ[t/ε] − I )ϕ1  ≤ 4C1 (ϕ1 R0 ),

(6.39)

.

as .Φkε,u are contractive operators. We note also that      t [t/ε]    ε,u  ˆ ˆ ¯ LΦk ϕ − LΦ(s)ϕds] . Eπ ×π1 [ε  ≤ εC2 (t, ϕ),  0   k=0

(6.40)

    D1 (·, u)ϕ , .t ∈ [0, T ]. This follows where .C2 (t, ϕ) := 4T E U π(dz)π1 (du) P from standard argument about the convergence of Riemann sums in Bochner integral (see Lemma 4.14, p. 161, [91]). We note that ε,u ε,u ε ϕ − Φ(t)ϕ|| + εC1 (ϕ1 R0 ), ||Eπ ×π1 [Φ[t/ε] ϕ ] − Φ(t)ϕ|| ≤ ||Eπ ×π1 [Φ[t/ε]

.

(6.41) where we applied representation .ϕ ε = ϕ + εϕ1 . ¯ We also note that .Φ(t) satisfies the equation ¯ −ϕ− .Φ(t)ϕ



t

ˆ Φ(s)ϕds ¯ L = 0.

(6.42)

0

Let us introduce the following martingale: ε,u ε .M [t/ε]+1 ϕ

:=

ε,u ε Φ[t/ε] ϕ

−ϕ − ε

[t/ε] 

ε,u ε Eπ ×π1 [Φk+1 ϕ − Φkε,u ϕ ε | Fku ].

(6.43)

k=0

This is of zero-mean-value martingale ε,u ε Eπ ×π1 M[t/ε] ϕ = 0,

(6.44)

.

which comes directly from (6.43). Again, from (6.43), we get the following asymptotic representation: ε,u ε ε,u M[t/ε] ϕ = Φ[t/ε] ϕ − ϕ + ε[Φ[t/ε] − I ]ϕ1 − ε

[t/ε] 

.

ˆ ε,u ϕ LΦ k

k=0

−ε2

[t/ε] 

D1 (·, u)Φ ε,u ϕ1 + oϕ (1)], [P k

k=0

(6.45)

136

6 Controlled Discrete-Time Semi-Markov Random Evolutions

where .oϕ (1) → 0, as .ε → 0, for any .ϕ ∈ B0 . Now, from Equation (6.4) and expressions (6.44) and (6.45), we obtain the following representation: ε,u ¯ Eπ ×π1 [Φ[t/ε] ϕ − Φ(t)ϕ] = εEπ ×π1 [Φ[t/ε]ε,u − I ]ϕ1 + Eπ ×π1 [ε

[t/ε] 

.

ˆ ε,u ϕ LΦ k

k=0



t

− 0

ˆ Φ(s)ϕds] ¯ L + ε2 Eπ ×π1 [

[t/ε]−1 

Rku (ϕ1 )],

k=0

(6.46) D1 (·, u)Φ ε,u ϕ1 + oϕ (1). where .Rku (ϕ1 ) := P  k  Let us estimate .Rku (ϕ1 ) in (6.46). .

      u R (ϕ1 ) ≤ sup (P D1 (z, u)g  + og (1)) := C2 (z, g, KΔ , u), k

(6.47)

g∈KΔ

where .KΔ is a compact set, .Δ > 0, because .Φkε,u ϕ1 satisfies compactness condition for any .ε > 0 and any k. In this way, we get from (6.46) that      [t/ε]−1    u   . Eπ ×π1 [ Rk (ϕ1 )] ≤ T π(dz)π1 (du)C3 (z, g, KΔ , u),  E   k=0

t ∈ [0, T ]. (6.48)

Finally, from inequalities (6.38)–(6.41) and from (6.47)–(6.48), we obtain the desired rate of convergence of the CDTSMRE in averaging scheme ε,u ε ||Eπ ×π1 [Φ[t/ε] ϕ ] − Φ(t)ϕ|| ≤ εA(T , ϕ, R0  , D1 (z, u)),

.

where the constant A(T , ϕ, R0  , D1 (z, u)) := 5C1 (ϕ, R0 ) + C2 (T , ϕ)  +T π(dz)π1 (du)C3 (z, g, KΔ , u), (6.49)

.

E

and .C3 (z, g, KΔ , u) is defined in (6.48). Therefore, the proof of Proposition 6.1 is done. Remark 6.3 In a similar way, we can obtain the rate of convergence results in diffusion approximation (see Propositions 6.2–6.3).

6.8 Concluding Remarks

137

6.8 Concluding Remarks In this chapter, we introduced controlled semi-Markov random evolutions in discrete-time in Banach space. The main results concerned time-rescaled limit theorems, namely, averaging, diffusion approximation, and diffusion approximation with equilibrium by martingale weak convergence method. We applied these results to various important families of stochastic systems, i.e., the controlled additive functionals, controlled geometric Markov renewal processes, and controlled dynamical systems. We provided dynamical principles for discrete-time dynamical systems such as controlled additive functionals and controlled geometric Markov renewal processes. We also produced dynamic programming equations (Hamilton–Jacobi– Bellman equations) for the limiting processes in diffusion approximation such as CAF, CGMRP, and CDS. As an example, we considered the solution of portfolio optimization problem by Merton for the limiting CGMRP in DA. We also point out the importance of convergence rates and obtained them in the limit theorems for CDTSMRE and CAF, CGMRP, and CDS. Extensions of the above results may be associated with the study of optimal control for the initial, not limiting models, such as CAF in Sect. 6.4.1, CGMRP in Sect. 6.4.2, and CDS in Sect. 6.4.3. Other optimal control problems would be also interesting to consider for diffusion models with equilibrium, e.g., CAF in Sect. 6.4.1 and CGMRP in Sect. 6.4.2. The latter models may be considered for solutions of Merton portfolio’s problems as well. Of course a main direction could be the case of dependent SMC .zk and the MC .uk . For further reading on this chapter the following references are of interest [49, 95, 164].

Chapter 7

Epidemic Models in Random Media

7.1 Introduction In the present chapter, we consider epidemic models in random semi-Markov environment in discrete time. For example, severe acute respiratory syndrome (SARS) is a viral respiratory disease of zoonotic origin caused by the SARS coronavirus (SARS-CoV). Similar examples are HIV, COVID-19, etc. Let us denote by .S(t) and .I (t) the number of susceptible and the number of infected individuals in a fixed population of N individuals, and the rate of transmission of the disease will be .βS(t)I (t)/N at time t. For a short interval, we consider that N is fixed. In fact, .S(t)I (t) is the number of contacts between infected and susceptible individuals. Let us also denote by .R(t) the number of removed individuals up to time t. The rate of removed individuals is .γ I (t)/N. The removed individuals include immunes, deaths, and isolates. The coefficients .β > 0 and .γ > 0 are fixed. The deterministic model proposed by [138], known as SIR model, with the above notation, is the following (see, e.g., [33]): ⎧ dS ⎨ dt = −βS(t)I (t) dI . dt = βS(t)I (t) − γ I (t) ⎩ dR dt = γ I (t) for .t ≥ 0, with .S(0) = n, .I (0) = m, .R(0) = r, .n + m + r = N , and .S(t) + I (t) + R(t) = N. This model is often represented by the following graph, and it is called a SIR model.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0_7

139

140

7 Epidemic Models in Random Media

SIR Model S(t)

β

I(t)

γ

R(t)

Of course, such a model may be considered with more than three states. For example, we can consider the additional states: Exposed, Hospitalized, etc., see next section. The above model can also be considered in discrete time as follows: ⎧ ⎨ Sk+1 = (1 − βIk )Sk . I = (1 + βSk − γ )Ik ⎩ k+1 Rk+1 = Rk + γ Ik for .k ∈ N, with .S0 = n, .I0 = m, .R0 = r, and .n + m + r = N . Under this condition of fixed population, N, the third equation above can be obtained as a combination of the first two equations. So we can replace one of the above equations by .Sk + Ik + Rk = N for any .k ∈ N. The condition of fixed population is valuable for a relatively short period of time. It can also be considered in random media or random environment expressed by a stochastic process, say .(zk , k ≥ 0). In that case, the coefficients can be considered as functions of .(zk ), i.e., .β = β(zk ), .γ = γ (zk ), and the sequences .Sk , .Ik and .Rk become random sequences. In the last case, we have finally a stochastic dynamical system switched by the stochastic process .(zk ). If the stochastic process .(zk ) is rapidly changing, compared with the .Sk , Ik , Rk , then we can consider it in rescaled time, i.e., .(z[t/εδ ] ), where .t ∈ R+ , .ε > 0, and .[·] means the integer part of the corresponding real number, and looking for limits when .ε → 0. Usually, .δ is equal to 1 or 2. This is the problem that we consider in the present chapter. We apply averaging, diffusion, and merging approximation theorems from previous Chaps. 3 and 4 to stochastic SARS model in semi-Markov random environment in discrete time.

7.2 From the Deterministic to Stochastic SARS Model The model presented here consists of the following compartments (states), see [176]: • Susceptibles S: Individuals not yet infected. • Exposed E: Susceptibles who have become infected and are not yet infectious. • Infectives I : Exposed individuals who have become infected and can spread the SARS coronavirus.

7.2 From the Deterministic to Stochastic SARS Model

141

• Hospitalized U : Infectives who are in the immediate environment of Health-Care Workers and Patients (HCWP); these individuals are not considered to pose any risk to the general public but may infect HCWP. • Removed R: Individuals who have been either exposed or infective, and who are considered to no longer be susceptible. The model can be represented by the following figure. ag bg S

E ah au

I

bh rg

rh cg

ch U eg

eh R

Thus, the model consists of eight coupled nonlinear difference equations describing the transfer of individuals from one compartment to another. The deterministic SARS model, in discrete time, has the following look: ⎧ g ⎪ Sk+1 ⎪ ⎪ ⎪ h ⎪ S ⎪ k+1 ⎪ ⎪ g ⎪ Ek+1 ⎪ ⎪ ⎪ ⎪ ⎪ Eh ⎪ ⎪ ⎨ gk+1 Ik+1 . h ⎪ Ik+1 ⎪ ⎪ g ⎪ ⎪ Uk+1 ⎪ ⎪ ⎪ h ⎪ ⎪ ⎪ Uk+1 ⎪ g ⎪ ⎪ ⎪ S0 ⎪ ⎩ g I0

= = = = = = = = = =

g

g

g

Sk − ag Sk (Ik + Ikh ) g g Skh − ah Skh (Ik + Ikh ) − au Skh (Ukh + Uk ) g g g g h Ek + ag Sk (Ik + Ik ) − bg Ek g g Ekh + ah Skh (Ik + Ikh ) + au Skh (Ukh + Uk ) − bh Ekh g g g g Ik + bg Ek − cg Ik − rg Ik h h h h Ik + bh Ek − ch Ik − rh Ik g g g Uk + rg Ik − eg Uk Ukh + rh Ikh − eh Ukh g Sg , S0h = Sh , E0 = Eg , E0h = Eh g Ig , I0h = Ih , U0 = Ug , U0h = Uh .

(7.1)

Here: .ag , ah , au are the transmission coefficients for the general public and HCWP infected, and of hospitalized infected for HCWP, respectively; .bg and .bh are the transmission coefficients of exposed individuals to the infective class; .cg and .ch are the transmission coefficients of infective individuals to the removed class; .rg and .rh are the transmission coefficients of infectives to hospitalization; .eg and .eh are the transmission coefficients to the removed class, reflecting the effectiveness of treatments.

142

7 Epidemic Models in Random Media

The second equation in Model (7.1) describes the additional risk of HCWP resulting from their direct contact with SARS patients in the health-care setting [176]. Let us consider now a semi-Markov chain (.zk , .k ∈ N), with state space the measurable space .(E, E ), and corresponding Markov renewal chain .(xn , τn , .n ∈ N), where .(xn ) is the embedded Markov chain and .(τn ) are the jump times. Denote by .q(x, B, k) the corresponding semi-Markov kernel, that is, .q(x, B, k) := P(xn+1 ∈ B, τn+1 − τn = k | xn = x), with .x ∈ E, .B ∈ E , and .k ∈ N, for any .n ∈ N. Denote also by .π(B), .B ∈ E the stationary probability of .(zk ), and .π  (B × {l}), .B ∈ E , .l ∈ N, the stationary probability of .(zk , γk ), where .(γk ) is the backward recurrence time process defined by .γk := k − τνk , k ≥ 1, .γ0 = 0, and .τ0 = 0 (see Chap. 3). In fact, z here represents a value of the environment, and the coefficients in the Equations (7.1) are not any more constants, but now they are functions, .a : E → Θ, where .Θ is the set of possible values of the coefficients. That is, when the environment at time .k ∈ N takes value z, the coefficient value is .a(z), and in fact, each coefficient in (7.1) is replaced by a stochastic process .a(zk ), .k ∈ N, see (7.2). The stochastic SARS model in semi-Markov random media has the following look: ⎧ g g g g Sk+1 = Sk − ag (zk )Sk (Ik + Ikh ) ⎪ ⎪ ⎪ g g ⎪ h ⎪ Sk+1 = Skh − ah (zk )Skh (Ik + Ikh ) − au (zk )Skh (Ukh + Uk ) ⎪ ⎪ ⎪ g g g g g ⎪ = Ek + ag (zk )Sk (Ik + Ikh ) − bg (zk )Ek E ⎪ ⎪ ⎪ k+1 g g ⎪ h h h h h ⎪ au (zk )Sk (Ukh + Uk ) − bh (zk )Ekh ⎪ ⎪ Egk+1 = Egk + ah (zk )Skg (Ik + Ik ) + ⎨ g g Ik+1 = Ik + bg (zk )Ek − cg (zk )Ik − rg (zk )Ik . , (7.2) h ⎪ Ik+1 = Ikh + bh (zk )Ekh − ch (zk )Ikh − rh (zk )Ikh ⎪ ⎪ g g g g ⎪ ⎪ Uk+1 = Uk + rg (zk )Ik − eg (zk )Uk ⎪ ⎪ ⎪ h ⎪ ⎪ Uk+1 = Ukh + rh (zk )Ikh − eh (zk )Ukh ⎪ ⎪ g g ⎪ ⎪ S = Sg , S0h = Sh , E0 = Eg , E0h = Eh ⎪ ⎪ ⎩ g0 g h I0 = Ig , I0 = Ih , U0 = Ug , U0h = Uh where functions .ag (z), ah (z), au (z), bg (z), bh (z), cg (z), ch (z), rg (z), rh (z), eg (z), eh (z), .z ∈ E, are continuous and bounded on E. Therefore, we suppose that our coefficients are random, not constants, and, in general, are functions of some random process, in our case, semi-Markov random process. The above equations can be written in the following vector form: Yk+1 = Yk + g(Yk , zk ),

.

(7.3)

7.3 Averaging of Stochastic SARS Models

143

where .g is an 8-dimensional vector-valued function defined on .R8 × E, which the eight components, .g(y, z) = [g1 (y, z), . . . , g8 (y, z)], are given by ⎧ g1 (y, z) ⎪ ⎪ ⎪ ⎪ ⎪ g2 (y, z) ⎪ ⎪ ⎪ ⎪ g3 (y, z) ⎪ ⎪ ⎨ g4 (y, z) . ⎪ g5 (y, z) ⎪ ⎪ ⎪ ⎪ g6 (y, z) ⎪ ⎪ ⎪ ⎪ g (y, z) ⎪ ⎪ ⎩ 7 g8 (y, z)

= = = = = = = =

−ag (z)Sg (Ig + Ih ) −ah (z)Sh (Ig + Ih ) − au (z)Sh (Uh + Ug ) ag (z)Sg (Ig + Ih ) − bg (z)Eg ah (z)Sh (Ig + Ih ) + au (z)Sh (Uh + Ug ) − bh (z)Eh bg (z)Eg − cg (z)Ig − rg (z)Ig bh (z)Eh − ch (z)Ih − rh (z)Ih rg (z)Ig − eg (z)Ug rh (z)Ih − eh (z)Uh ,

(7.4)

where .y = [S g , S h , E g , E h , I g , I h , U g , U h ] ∈ R8 and .z ∈ E. We will study in this chapter the above stochastic SARS model (7.2) in semiMarkov random media.

7.3 Averaging of Stochastic SARS Models We consider the stochastic SARS model in series scheme, where the rescaled time is .k := [t/ε] ∈ N, for .t ∈ R+ , .ε > 0, the small series parameter. It takes the following form: ⎧ g,ε St ⎪ ⎪ ⎪ ⎪ ⎪ Sth,ε ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ g,ε ⎪ ⎪ Et ⎪ ⎪ ⎪ ⎪ ⎪ Eth,ε ⎪ ⎪ ⎨ .

g,ε ⎪ It ⎪ ⎪ ⎪ ⎪ ⎪ Ith,ε ⎪ ⎪ ⎪ g,ε ⎪ Ut ⎪ ⎪ ⎪ ⎪ h,ε ⎪ ⎪ Ut ⎪ ⎪ ⎪ ⎪ S0g,ε ⎪ ⎪ ⎩ g,ε I0

g

g

g

h ) = S[t/ε] − ag (z[t/ε] )S[t/ε] (I[t/ε] + I[t/ε] g h h h = S[t/ε] − ah (z[t/ε] )S[t/ε] (I[t/ε] + I[t/ε] ) g h (U h −au (z[t/ε] )S[t/ε] [t/ε] + U[t/ε] ) g g g g h ) − b (z = E[t/ε] + ag (z[t/ε] )S[t/ε] (I[t/ε] + I[t/ε] g [t/ε] )E[t/ε] h h (I g h = E[t/ε] + ah (z[t/ε] )S[t/ε] [t/ε] + I[t/ε] ) g h h h +au (z[t/ε] )S[t/ε] (U[t/ε] + U[t/ε] ) − bh (z[t/ε] )E[t/ε] g g g g = Ik + bg (zk )Ek − cg (zk )Ik − rg (zk )Ik h h h h = I[t/ε] + bh (z[t/ε] )E[t/ε] − ch (z[t/ε] )I[t/ε] − rh (z[t/ε] )I[t/ε] g g g = U[t/ε] + rg (z[t/ε] )I[t/ε] − eg (z[t/ε] )U[t/ε] h h h = U[t/ε] + rh (z[t/ε] )I[t/ε] − eh (z[t/ε] )U[t/ε] g,ε h,ε h,ε = Sg , S0 = Sh , E0 = Eg , E0 = Eh g,ε = Ig , I0h,ε = Ih , U0 = Ug , U0h,ε = Uh .

(7.5)

Let .g(y, z) := g(S g , S h , E g , E h , I g , I h , U g , U h ; z), and the vector function .g(y, z) whose components are given by (7.4), with .zk = z. We note that in this case .y = [S g , S h , E g , E h , I g , I h , U g , U h ] ∈ R8 .

144

7 Epidemic Models in Random Media

Then, from Proposition 4.10 in Chap. 4, it follows that g,ε St ⇒  Sg (t), Sth,ε ⇒  Sh (t), as ε g,ε It ⇒ Ig (t), Ith,ε ⇒ Ih (t), as ε . g,ε g (t), Eth,ε ⇒ E h (t), as ε Et ⇒ E g,ε h,ε g (t), Ut ⇒ U h (t), as ε Ut ⇒ U

→ 0, → 0, → 0, → 0,

(7.6)

where ⎧ d ag  Sg (t)(Ig (t) + Ih (t)) ⎪ dt Sg (t) = − ⎪ ⎪ d ⎪ h (t) + U g (t))  ah Sh (t)(Ig (t) + Ih (t)) −  au Sh (t)(U ⎪ dt Sg (t) = − ⎪ ⎪ d  ⎪ g (t) ⎪ ag  bg E Sg (t)(Ig (t) + Ih (t)) −  ⎪ dt Eg (t) =  ⎪ ⎪ d    h (t) + U g (t)) −    h (t) ⎪ (t) =  a (t)( I (t) + I (t)) +  a bh E E S S h h g h u h (t)(U ⎪ dt h ⎪ ⎨ d     cg Ig (t) −  rg Ig (t) dt Ig (t) = bg Eg (t) −  . (7.7) d  h (t) −  ⎪ dt bh E ch Ih (t) −  rh Ih (t) Ih (t) =  ⎪ ⎪ ⎪ d  g (t) ⎪ rg Ig (t) −  eg U ⎪ dt Ug (t) =  ⎪ ⎪ d  ⎪   (t) =  r (t) −  e U I U ⎪ h h h h (t) ⎪ dt h ⎪ ⎪  g (0) = Eg , E h (0) = Eh  ⎪ S (0) = Sg , Sh (0) = Sh , E ⎪ ⎩ g     Ig (0) = Ig , Ih (0) = Ih , Ug (0) = Ug , Uh (0) = Uh and  ag  ah  au  bg  bh . cg  ch  rg  rh  eg  eh

:= := := := := := := := := := :=

 E π(dy)ag (y), E π(dy)ah (y), E π(dy)au (y), E π(dy)bg (y), E π(dy)bh (y), E π(dy)cg (y), E π(dy)ch (y), E π(dy)rg (y), E π(dy)rh (y), E π(dy)eg (y), E π(dy)eh (y),

(7.8)

where .π is the stationary probability distribution of the SMC .(zk ), defined in Chap. 2. The system of equations (7.7) is a deterministic system and in fact a Cauchy initial value problem.

7.4 SARS Model in Merging Semi-Markov Random Media

145

7.4 SARS Model in Merging Semi-Markov Random Media The switching process .(zk ) can be simplified by asymptotic methods under certain hypothesis. In that case, we obtain a switched system with a much simpler switching process (see Sect. 2.6.1). In such a simplification, we have to pay attention in order that the new switching process retains the essential characteristics of the environment. This generalizes the previous section where the limit switching process is reduced to one state, that is, in a fixed environment. Let us consider the following split of the state space E: E = ∪dj =1 Ej ,

Ei ∩ Ej = Ø,

.

i = j.

(7.9)

The semi-Markov kernels on the split space have the following representation: q ε (x, B, k) = P ε (x, B)fx (k),

.

(7.10)

where the transition kernel of the EMC, .(xnε , n ≥ 0), has the representation P ε (x, B) = P (x, B) + εP1 (x, B).

.

(7.11)

The transition kernel P determines a support Markov chain and satisfies the following relations:  P (x, Ej ) = 1j (x) =

.

1 if x ∈ Ej 0 if x ∈ Ej ,

(7.12)

for .j = 1, . . . , d. Of course, the signed perturbing kernel .P1 satisfies the relation P1 (x, E) = 0, and .P ε (x, E) = P (x, E) = 1. We consider the following stochastic SARS model in series scheme:

.

⎧ g,ε St ⎪ ⎪ ⎪ ⎪ h,ε ⎪ ⎪ St ⎪ ⎪ g,ε ⎪ ⎪ Et ⎪ ⎪ ⎪ ⎪ ⎪ Eth,ε ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ .

g,ε

It ⎪ ⎪ ⎪ ⎪ Ith,ε ⎪ ⎪ ⎪ g,ε ⎪ Ut ⎪ ⎪ ⎪ ⎪ ⎪ Uth,ε ⎪ ⎪ ⎪ ⎪ S g,ε ⎪ ⎪ 0 ⎪ ⎩ g,ε I0

= = = = = = = = = =

g,ε

g,ε

g,ε

h,ε ε S[t/ε] − ag (z[t/ε] )S[t/ε] (I[t/ε] + I[t/ε] ) g,ε g,ε h,ε h,ε h,ε h,ε h,ε ε ε S[t/ε] − ah (z[t/ε] )S[t/ε] (I[t/ε] + I[t/ε] ) − au (z[t/ε] )S[t/ε] (U[t/ε] + U[t/ε] ) g,ε g,ε g,ε g,ε h,ε ε ε E[t/ε] + ag (z[t/ε] )S[t/ε] (I[t/ε] + I[t/ε] ) − bg (z[t/ε] )E[t/ε] g,ε g,ε g,ε h,ε h,ε h,ε h,ε ε ε E[t/ε] + ah (z[t/ε] )S[t/ε] (I[t/ε] + I[t/ε] ) + au (z[t/ε] )S[t/ε] (U[t/ε] + U[t/ε] ) h,ε ε −bh (z[t/ε] )E[t/ε] g,ε g,ε g,ε ε (7.13) , I[t/ε] + bg (z[ t/ε]ε )E[t/ε] − cg (z[t/ε] )I[t/ε] − rg (z[ t/ε]ε )I[ t/ε]g,ε h,ε h,ε h,ε h,ε ε ε ε I[t/ε] + bh (z[t/ε] )E[t/ε] − ch (z[t/ε] )I[t/ε] − rh (z[t/ε] )I[t/ε] g,ε g,ε g,ε ε ε U[t/ε] + rg (z[t/ε] )I[t/ε] − eg (z[t/ε] )U[t/ε] h,ε h,ε h,ε ε ε U[t/ε] + rh (z[t/ε] )I[t/ε] − eh (z[t/ε] )U[t/ε] g,ε h,ε Sg , S0 = Sh , E0 = Eg , E0h,ε = Eh g,ε Ig , I0h,ε = Ih , U0 = Ug , U0h,ε = Uh

ε where .(z[t/ε] ) is the perturbing ergodic semi-Markov process.

146

7 Epidemic Models in Random Media

We note that in this case .y = [S g , S h , E g , E h , I g , I h , U g , U h ] ∈ R8 . Let .g(y, z) := g(S g , S h , E g , E h , I g , I h , U g , U h ; z), .z ∈ E, and the function .g(y, z) is given in (7.4). We now use Theorem 2.5, in Chap. 2, with a vector function .g(x, z), from which it follows that g,ε St ⇒ Sg (t), Sth,ε ⇒ Sh (t), as ε g,ε It ⇒ I g (t), Ith,ε ⇒ I h (t), as ε . g,ε g (t), Eth,ε ⇒ E h (t), as ε Et ⇒ E g,ε h,ε g (t), Ut ⇒ U h (t), as ε Ut ⇒ U

→ 0, → 0, → 0, → 0,

(7.14)

where the limit processes are solution of the following perturbed Cauchy problem. The perturbing process here .( yt , t ∈ R+ ) is much simpler than the initial perturbing ε process .z[t/ε] . ⎧ d ⎪ ⎪ dt Sg (t) ⎪ ⎪ d ⎪ ⎪ dt Sh (t) ⎪ ⎪ ⎪ dE g (t) ⎪ ⎪ ⎪ dt ⎪ d ⎪ E h (t) ⎪ ⎪ ⎨ dt d I dt g (t) . d ⎪ ⎪ dt Ih (t) ⎪ ⎪ d ⎪ ⎪ dt U g (t) ⎪ ⎪ d ⎪ ⎪ dt Uh (t) ⎪ ⎪ ⎪ ⎪ Sg (0) ⎪ ⎪ ⎩ Ig (0)

= − ag ( yt ) Sg (t)(I g (t) + I h (t)) h (t) + U g (t)) = − ah ( yt ) Sh (t)(I g (t) + I h (t)) − au ( yt ) Sh (t)(U g (t) = ag ( yt ) Sg (t)(I g (t) + I h (t)) − bg ( y (t))E h (t) + U g (t)) − h (t) = ah ( yt ) Sh (t)(I g (t) + I h (t)) + au ( yt ) Sh (t)(U bh ( yt )E g (t) − = bg ( y t )E cg ( yt )I g (t) − rg ( yt )I g (t) (7.15) h (t) − = bh ( yt )E ch ( yt )I h (t) − rh ( yt )I h (t) g (t) = rg ( yt )I g (t) − eg ( y t )U = rh ( yt )Ih (t) − eh ( yt )Uh (t) g (0) = Eg , E h (0) = Eh = Sg , Sh (0) = Sh , E = Ig , Ih (0) = Ih , Ug (0) = Ug , Uh (0) = Uh .

The coefficient functions of the equations (7.15), defined on the split and merged  see Sect. 2.6.1, are given by state space, .E, ag (v) ah (v) au (v) bg (v) bh (v) cg (v) . ch (v) rg (v) rh (v) eg (v) eh (v)

:= := := := := := := := := := :=

 π (dy)ag (y), E v v π (dy)ah (y), E v v π (dy)au (y), E v v π (dy)bg (y), E v v π (dy)bh (y), E v v π (dy)cg (y), E v v π (dy)ch (y), E v v π (dy)rg (y), E v v π (dy)rh (y), E v v π (dy)eg (y), E v v Ev πv (dy)eh (y),

(7.16)

7.5 Diffusion Approximation of Stochastic SARS Models in Semi-Markov. . .

147

where .Ev , and .πv are defined in Chap. 2 and . yt is a merged Markov process, see Sect. 2.6.1. The merged process . xt in Theorem 2.5 is denoted here by . yt .

7.5 Diffusion Approximation of Stochastic SARS Models in Semi-Markov Random Media We now suppose that all the averaged transmission coefficients in (7.8) are equal to zero. It means that we have a balance condition for the averaged stochastic SARS model in (7.7). We may consider as coefficients .a(x) − a , from (7.8), in the place of .a(x), for which we have the balance condition .Π (a(x) −  a ) = 0 as required. Let us consider the following stochastic SARS model in series scheme and scaling time .k = [t/ε2 ] : ⎧ g,ε St ⎪ ⎪ ⎪ ⎪ 2 ⎪ ⎪ Sth,ε ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ g,ε ⎪ Et ⎪ ⎪ ⎪ ⎪ h,ε ⎪ ⎪ E ⎪ ⎪ t ⎨ .

g,ε ⎪ ⎪ It ⎪ ⎪ ⎪ ⎪ I h,ε ⎪ t ⎪ ⎪ ⎪ g,ε ⎪ ⎪ U t ⎪ ⎪ ⎪ h,ε ⎪ ⎪ U ⎪ t ⎪ ⎪ g,ε ⎪ ⎪ S0 ⎪ ⎪ ⎩ g,ε I0

g

g

g

h = S[t/ε2 ] − εag (z[t/ε2 ] )S[t/ε2 ] (I[t/ε2 ] + I[t/ε 2]) g

h h h = S[t/ε 2 ] − εah (z[t/ε 2 ] )S[t/ε 2 ] (I[t/ε 2 ] + I[t/ε 2 ] ) g

h h −εau (z[t/ε2 ] )S[t/ε 2 ] (U[t/ε 2 ] + U[t/ε 2 ] ) g

g

g

g

h = E[t/ε2 ] + εag (z[t/ε2 ] )S[t/ε2 ] (I[t/ε2 ] + I[t/ε 2 ] ) − εbg (z[t/ε 2 ] )E[t/ε 2 ] g

h h h = E[t/ε 2 ] + ah (z[t/ε 2 ] )S[t/ε 2 ] (I[t/ε 2 ] + I[t/ε 2 ] ) g

h h h +εau (z[t/ε2 ] )S[t/ε 2 ] (U[t/ε 2 ] + U[t/ε 2 ] ) − εbh (z[t/ε 2 ] )E[t/ε 2 ]

(7.17) g g g = Ik + bg (z[ t/ε2 ])Ek − εcg (z[ t/ε2 ])I[ t/ε2 ]g − εrg (z[ t/ε2 ])Ik h h h h = I[t/ε 2 ] + εbh (z[t/ε 2 ] )E[t/ε 2 ] − εch (z[t/ε 2 ] )I[t/ε 2 ] − rh (z[t/ε 2 ] )I[t/ε 2 ] g

g

g

= U[t/ε2 ] + εrg (z[t/ε2 ] )I[t/ε2 ] − εeg (z[t/ε2 ] )U[t/ε2 ] h h h = U[t/ε 2 ] + εrh (z[t/ε 2 ] )I[t/ε 2 ] − εeh (z[t/ε 2 ] )U[t/ε 2 ]

= Sg , = Ig ,

g,ε

S0h,ε = Sh , E0 = Eg , E0h,ε = Eh , g,ε I0h,ε = Ih , U0 = Ug , U0h,ε = Uh .

It is worth noticing that in this case .y = [S g , S h , E g , E h , I g , I h , U g , U h ] ∈

R8 .

Let .g(y, z) := g(S g , S h , E g , E h , I g , I h , U g , U h ; z), and the vector-valued function .g(y, z) is equal to the right-hand side of the system (7.17), with .z[t/ε2 ] = z. The following assumptions have to be satisfied:  •  .g(y) := E π(dz)g(y, z) = 0.  • . E π(dz)g  (y, z)g(y, z) < ∞. 



• . E π(dz) ∂y∂ j gi (y, z) < ∞, for any .y ∈ R8 and .1 ≤ i, j ≤ d. • The third moments of .fz (k), .z ∈ E, are uniformly integrable (see Chap. 2).

148

7 Epidemic Models in Random Media

Then, from Proposition 5.5, in Chap. 5, together with the above assumptions, it follows that g,ε

St ⇒ Sg (t), Sth,ε ⇒ Sh (t), as ε g,ε It ⇒ Ig (t), Ith,ε ⇒ Ih (t), as ε . g,ε Et ⇒ Eg (t), Eth,ε ⇒ Eh (t), as ε g,ε Ut ⇒ Ug (t), Uth,ε ⇒ Uh (t), as ε

→ 0, → 0, → 0, → 0,

(7.18)

that is, Yε[t/ε2 ] ⇒ Y(t),

.

ε → 0,

Y(t) satisfies the following stochastic differential equation: where . d Y(t) = α ( Y(t))dt + β( Y(t))dw(t),

.

(7.19)

with Y(t) := [S g (t), S h (t), E g (t), E h (t), I g (t), I h (t), U g (t), U h (t)],

.

w(t) is 8-dimensional Wiener process whose components are standard Wiener processes. Here, the drift vector coefficient is as follows:  . α (y) := π  (dz, {k})g (y, z)(R0 − I )g y (y, z), (7.20)

.

k≥0 E

where .g y (y, z) is the Jacobian matrix, with respect to the variables .y = (y1 , . . . , y8 ). The above drift coefficient has the following look: g (t), E h (t), U g (t), U h (t)))8 ] S g (t),  S h (t), Ig (t), Ih (t), E α ( Y(t)) := [(αi ( i=1  g h g h g h       g (t), U h (t), z))8 ] . := E π(dz)[(αi (S (t),S (t), I (t), I (t), E (t), E (t), U i=1  8  := E π(dz)[(αi (z))i=1 ] ,

7.5 Diffusion Approximation of Stochastic SARS Models in Semi-Markov. . .

149

where α1 (y) := ag (y)R0 ag (y)S g (I g + I h )2 , α2 (y) := (−ah (y)Sh (I g + I h ) − au (y)S h (U g + U h ))R0 ag (y)(I g + I h ) +(−ag (y)S h (I g + I h ) − bg (y)E g )R0 ag (y)(I g + I h ) + (ah (y)S h (I g + I h ) +au (y)S h (U g + U h ) − bh (y)E h )R0 (ah (y)(I g + I h ) + au (y)(U g + U h )), α3 (y) := −(−ag (y)S h (I g + I h ) − bg (y)E g )R0 bg (y) +(bg (y)E g − cg (y)I g − rg (y)I g )R0 bg (y), α4 (y) := −(ah (y)S h (I g + I h ) + au (y)S h (U g + U h ) − bh (y)E h )R0 · bh (y) +(bh (y)E h − ch (y)I h − rh (y)I h )R0 bh (y), α5 (y) := ag (y)R0 ag (y)(S g )2 (I g + I h ) − (−ah (y)S h (I g + I h ) −au (y)S h (U g + U h ))R0 ah (y)S h + (−ag (y)S h (I g + I h ) −bg (y)E g )R0 ag (y)S h + (ah (y)S h (I g + I h ) + au (y)S h (U g + U h ) −bh (y)E h )R0 ah (y)S h − (bg (y)E g − cg (y)I g − rg (y)I g )R0 cg (y) . (7.21) +(rg (y)I g − eg (y)U g )R0 rg (y), α6 (y) := ag (y)R0 ag (y)(S g )2 (I g + I h ) − (−ah (y)S h (I g + I h ) −au (y)S h (U g + U h ))R0 ah (y)S h + (−ag (y)S h (I g + I h ) −bg (y)E g )R0 ag (y)S h + (ah (y)S h (I g + I h ) + au (y)S h (U g + U h ) −bh (y)E h )R0 ah (y)S h + (bh (y)E h − ch (y)I h − rh (y)I h )R0 − ch (y) − rh (y)) +(rh (y)I h − eh (y)U h )R0 rh (y), α7 (y) := −(−ah (y)S h (I g + I h ) − au (y)S h (U g + U h ))R0 au (y)S h +(ah (y)S h (I g + I h ) + au (y)S h (U g + U h ) − bh (y)E h )R0 au (y)S h −(rg (y)I g − eg (y)U g )R0 eg (y), α8 (y) := −(−ah (y)S h (I g + I h ) − au (y)S h (U g + U h ))R0 au (y)S h +(ah (y)S h (I g + I h ) + au (y)S h (U g + U h ) − bh (y)E h )R0 au (y)S h −(rh (y)I h − eh (y)U h )R0 eh (y),

and the diffusion matrix in (7.19) is given by β 2 (y) := 2



.

k≥0

1 π  (dx, {k})g (y, x)(R0 − I )g(y, x), 2 E

(7.22)

which takes the following form: ij ; i, j = 1, 2, 3, . . . , 8), β( Y(t)) := (β

.

where 2 := β 2 (S g , S h , E g , E h , I g , I h , U g , U h ) β ij ij  := 2 E π(dx)[βij1 (S g , S h , E g , E h , I g , I h , U g , U h , x) . +βij2 (S g , S h , E g , E h , I g , I h , U g , U h , x)]  := 2 E π(dx)[βij1 (x) + βij2 (x)],

(7.23)

150

7 Epidemic Models in Random Media

where coefficients .βijk (y), .k = 1, 2, .i, j = 1, 2, . . . , 8, are the following: 1

.β11 (y)

:= ag (y) · R0 · ag (y) · (S g )2 (I g + I h )2 ,

1 β12 (y) := −ag (y) · S g · (I g + I h ) · R0 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )), 1 (y) := −ag (y) · S g · (I g + I h ) · R0 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ), β13 1 β14 (y) := −ag (y) · S g · (I g + I h ) · R0 · (ah (y) · S h · (I g + I h )

+au (y) · S h · (U g + U h ) − bh (y) · E h ), 1 β15 (y) := −ag (y) · S g · (I g + I h ) · R0 · (bg (y)E g − cg (y)I g − rg (y)I g ), 1 β16 (y) := −ag (y) · S g · (I g + I h ) · R0 · (bh (y)E h − ch (y)I h − rh (y)I h ), 1 (y) := −ag (y) · S g · (I g + I h ) · R0 · (rg (y)I g − eg (y)U g ), β17 1 β18 (y) := −ag (y) · S g · (I g + I h ) · R0 · (rh (y)I h − eh (y)U h ), 1 (y) := −ag (y) · S g · (I g + I h ) · R0 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )), β21 1 β22 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (−ah (y) · S h · (I g + I h )

−au (y) · S h · (U g + U h )), 1 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (−ag (y) · S h · (I g + I h ) β23

−bg (y) · E g ), 1 β24 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (ah (y) · S h · (I g + I h )

+au (y) · S h · (U g + U h ) − bh(y) · E h ), 1 β25 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (bg (y) · E g − cg (y) · I g

−rg (y) · I g ), 1 β26 (y) := (−ah (y) · S h · (Ig + I h ) − au (y) · S h · (U g + U h )) · R0 · (bh (y) · E h − ch (y) · I h

−rh (y) · I h ), 1 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (rg (y) · I g − eg (y) · U g ), β27 1 β28 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (rh (y) · I h − eh (y) · U h ), 1 β31 (y) := −ag (y) · S g · (I g + I h ) · R0 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ), 1 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (−ag (y) · S h · (I g + I h ) β32

−bg (y) · E g ),

7.5 Diffusion Approximation of Stochastic SARS Models in Semi-Markov. . . 1

.β33 (y)

151

:= (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · R0 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ),

1 β34 (y) := (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · R0 · (ah (y) · S h · (I g + I h )

+au (y) · S h · (U g + U h ) − bh (y) · E h ), 1 (y) := (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · R0 · (bg (y) · E g − cg (y) · I g − rg (y) · I g ), β35 1 β36 (y) := (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · R0 · (bh (y) · E h − ch(y) · I h − rh (y) · I h ), 1 (y) := (−ag (y) · Sh · (Ig + I h) − bg (y) · Eg) · R0 · (rg (y) · Ig − eg (y) · Ug), β37 1 (y) := (−ag (y) · S h · (Ig + I h ) − bg (y) · E g ) · R0 · (rh (y) · I h − eh (y) · U h )], β38 1 (y) := −ag (y) · S g · (I g + I h ) · R0 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) β41

−bh (y) · E h ), 1 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (ah (y) · S h · (I g + I h ) β42

+au (y) · S h · (U g + U h ) − bh (y) · E h ), 1 β43 (y) := (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · R0 · (ah (y) · S h · (I g + I h )

+au (y) · S h · (U g + U h ) − bh (y) · E h ), 1 β44 (y) := (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h )

−bh (y) · E h ) · R0 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ), 1 β45 (y) := (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) · R0 · (bg (y) · E g

−cg (y) · I g − rg (y) · I g ), 1 β46 (y) := (ah (y) · S h · (I g + I h ) + au(y) · S h · (U g + U h ) − bh (y) · E h ) · R0 · (bh (y) · E h

−ch (y) · I h − rh (y) · I h ), 1 (y) := (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) · R0 · (rg (y) · I g β47

−eg (y) · U g ), 1 (y) := (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) · R0 · (rh (y) · I h β48

−eh (y) · U h ), 1 β51 (y) := −ag (y) · S g · (I g + I h ) · R0 · (bg (y) · E g − cg (y) · I g − rg (y) · I g ), 1 β52 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (bg (y) · E g − cg (y) · I g

−rg (y) · I g ), 1 β53 (y) := (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · R0 · (bg (y) · E g − cg (y) · I g − rg (y) · I g ), 1 (y) := (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) · R0 · (bg (y) · E g β54

−cg (y) · I g − rg (y) · I g ), 1 (y) := (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · R0 · (bg (y) · E g − cg (y) · I g − rg (y) · I g ), β55 1 (y) := (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · R0 · (bh (y) · E h − ch (y) · I h − rh (y) · I h ), β56 1 β57 (y) := (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · R0 · (rg (y) · I g − eg (y) · U g ),

152

7 Epidemic Models in Random Media

1 β58 (y) := (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · R0 · (rh (y) · I h − eh (y) · U h ), 1 β61 (y) := −ag (y) · S g · (I g + I h ) · R0 · (bh (y) · E h − ch (y) · I h − rh (y) · I h ), 1 β62 (y) := (−ah (y) · S h · (I g + I h) − au (y) · S h · (U g + U h )) · R0 · (bh (y) · E h − ch (y) · I h

−rh (y) · I h ), 1 β63 (y) := (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · R0 · (bh (y) · E h − ch (y) · I h − rh (y) · I h ), 1 (y) := (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) · R0 · (bh (y) · E h β64

−ch (y) · I h − rh (y) · I h ), 1 (y) := (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · R0 · (bh (y) · E h − ch (y) · I h − rh (y) · I h ), β65 1 β66 (y) := (bh (y) · E h − ch (y) · I h − rh (y) · I h ) · R0 · (bh (y) · E h − ch (y) · I h − rh (y) · I h ), 1 (y) := (bh (y) · E h − ch (y) · I h − rh (y) · I h ) · R0 · (rg (y) · I g − eg (y) · U g ), β67 1 (y) := (bh (y) · E h − ch (y) · I h − rh (y) · I h ) · R0 · (rh (y) · I h − eh (y) · U h ), β68

1

.β71 (y)

:= −ag (y) · S g · (I g + I h ) · R0 · (rg (y) · I g − eg (y) · U g ),

1 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (rg (y) · I g − eg (y) · U g ), β72 1 β73 (y) := (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · R0 · (rg (y) · I g − eg (y) · U g ), 1 β74 (y) := (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) · R0 · (rg (y) · I g

−eg (y) · U g ), 1 β75 (y) := (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · R0 · (rg (y) · I g − eg (y) · U g ), 1 β76 (y) := (bh (y) · E h − ch (y) · I h − rh (y) · I h ) · R0 · (rg (y) · I g − eg (y) · U g ), 1 β77 (y) := (rg (y) · I g − eg (y) · U g ) · R0 · (rg (y) · I g − eg (y) · U g ), 1 β78 (y) := (rg (y) · I g − eg (y) · U g ) · R0 · (rh (y) · I h − eh (y) · U h ), 1 (y) := −ag (y) · S g · (I g + I h ) · R0 · (rh (y) · I h − eh (y) · U h ), β81 1 β82 (y) := (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · R0 · (rh (y) · I h − eh (y) · U h ), 1 β83 (y) := (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · R0 · (rh (y) · I h − eh (y) · U h ), 1 (y) := (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) · R0 · (rh (y) · I h β84

−eh (y) · U h ), 1 (y) := (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · R0 · (rh (y) · I h − eh (y) · U h ), β85 1 β86 (y) := (bh (y) · E h − ch (y) · I h − rh (y) · I h ) · R0 · (rh (y) · I h − eh (y) · U h ), 1 β87 (y) := (rg (y) · I g − eg (y) · U g ) · R0 · (rh (y) · I h − eh (y) · U h ), 1 (y) := (rh (y) · I h − eh (y) · U h ) · R0 · (rh (y) · I h − eh (y) · U h ), β88

7.5 Diffusion Approximation of Stochastic SARS Models in Semi-Markov. . . 2 β11 (y) := .5 · ag (y)2 · (S g )2 · (I g + I h )2 , 2 β12 (y) := −.5 · ag (y) · S g · (I g + I h ) · (−ah (y) · S h · (I g + I h )

−au (y) · S h · (U g + U h )), 2 (y) := −.5 · ag (y) · S g · (I g + I h ) · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ), β13 2 β14 (y) := −.5 · ag (y) · S g · (I g + I h ) · (ah (y) · S h · (I g + I h )

+au (y) · S h · (U g + U h ) − bh (y) · E h ), 2 β15 (y) := −.5 · ag (y) · S g · (I g + I h ) · (bg (y) · E g − cg (y) · I g − rg (y) · I g ), 2 β16 (y) := −.5 · ag (y) · S g · (I g + I h ) · (bh (y) · E h − ch (y) · I h − rh (y) · I h ), 2 β17 (y) := −.5 · ag (y) · S g · (I g + I h ) · (rg (y) · I g − eg (y) · U g ), 2 β18 (y) := −.5 · ag (y) · S g · (I g + I h ) · (rh (y) · I h − eh (y) · U h ), 2 β21 (y) := −.5 · ag (y) · S g · (I g + I h ) · (−ah (y) · S h · (I g + I h )

−au (y) · S h · (U g + U h )), 2

.β22 (y)

:= .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h ))2 ,

2 β23 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h ))

·(−ag (y) · S h · (I g + I h ) − bg (y) · E g ), 2 β24 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h ))

·(ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ), 2 β25 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · (bg (y) · E g

−cg (y) · I g − rg (y) · I g ), 2 β26 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · (bh (y) · E h

−ch (y) · I h − rh (y) · I h ), 2 β27 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · (rg (y) · I g

−eg (y) · U g ), 2 β28 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · (rh (y) · I h

−eh (y) · U h )], 2 β31 (y) := −.5 · ag (y) · S g · (I g + I h ) · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ), 2 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) β32

·(−ag (y) · S h · (I g + I h ) − bg (y) · E g ), 2 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g )2 , β33 2 β34 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (ah (y) · S h · (I g + I h )

+au (y) · S h · (U g + U h ) − bh (y) · E h ),

153

154

7 Epidemic Models in Random Media

2 β35 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (bg (y) · E g − cg (y) · I g

−rg (y) · I g ), 2 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (bh (y) · E h − ch (y) · I h β36

−rh (y) · I h ), 2 β37 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (rg (y) · I g − eg (y) · U g ), 2 β38 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (rh (y) · I h − eh (y) · U h ), 2 (y) := −.5 · ag (y) · S g · (I g + I h ) · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) β41

−bh (y) · E h ), 2 β42 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h ))

·(ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ), 2 β43 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (ah (y) · S h · (I g + I h )

+au (y) · S h · (U g + U h ) − bh (y) · E h ), 2 β44 (y) := .5 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h )2 , 2

.β45 (y)

:= .5 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) ·(bg (y) · E g − cg (y) · I g − rg (y) · I g ),

2 β46 (y) := .5 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h )

·(bh (y) · E h − ch (y) · I h − rh (y) · I h ), 2 β47 (y) := .5 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h )

·(rg (y) · I g − eg (y) · U g ), 2 (y) := .5 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) β48

·(rh (y) · I h − eh (y) · U h ), 2 β51 (y) := −.5 · ag (y) · S g · (I g + I h ) · (bg (y) · E g − cg (y) · I g − rg (y) · I g ), 2 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · (bg (y) · E g β52

−cg (y) · I g − rg (y) · I g ), 2 β53 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (bg (y) · E g − cg (y) · I g

−rg (y) · I g ), 2 β54 (y) := .5 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h )

·(bg (y) · E g − cg (y) · I g − rg (y) · I g ), 2 β55 (y) := .5 · (bg (y) · E g − cg (y) · I g − rg (y) · I g )2 , 2 β56 (y) := .5 · (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · (bh (y) · E h − ch (y) · I h

−rh (y) · I h ),

7.5 Diffusion Approximation of Stochastic SARS Models in Semi-Markov. . .

155

2 β57 (y) := .5 · (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · (rg (y) · I g − eg (y) · U g ), 2 (y) := .5 · (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · (rh (y) · I h − eh (y) · U h ), β58 2 β61 (y) := −.5 · ag (y) · S g · (I g + I h ) · (bh (y) · E h − ch (y) · I h − rh (y) · I h ), 2 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · (bh (y) · E h β62

−ch (y) · I h − rh (y) · I h ), 2 β63 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (bh (y) · E h − ch (y) · I h − rh (y) · I h ), 2 (y) := .5 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) β64

·(bh (y) · E h − ch (y) · I h − rh (y) · I h ), 2 β65 (y) := .5 · (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · (bh (y) · E h − ch (y) · I h − rh (y) · I h ), 2 (y) := .5 · (bh (y) · E h − ch (y) · I h − rh (y) · I h )2 , β66 2 β67 (y) := .5 · (bh (y) · E h − ch (y) · I h − rh (y) · I h ) · (rg (y) · I g − eg (y) · U g ), 2 (y) := .5 · (bh (y) · E h − ch (y) · I h − rh (y) · I h ) · (rh (y) · I h − eh (y) · U h ), β68 2 β71 (y) := −.5 · ag (y) · S g · (I g + I h ) · (rg (y) · I g − eg (y) · U g ), 2 β72 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · (rg (y) · I g − eg (y) · U g ), 2 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (rg (y) · I g − eg (y) · U g ), β73 2

.β74 (y)

:= .5 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) · (rg (y) · I g −eg (y) · U g ),

2 β75 (y) := .5 · (bg (y) · E g − cg (y) · I g − rg(y) · I g ) · (rg (y) · I g − eg (y) · U g ), 2 (y) := .5 · (bh (y) · E h − ch (y) · I h − rh(y) · I h ) · (rg (y) · I g − eg (y) · U g ), β76 2 β77 (y) := .5 · (rg (y) · I g − eg (y) · U g )2 , 2 β78 (y) := .5 · (rg (y) · I g − eg (y) · U g ) · (rh (y) · I h − eh (y) · U h ), 2 (y) := −.5 · ag (y) · S g · (I g + I h ) · (rh (y) · I h − eh (y) · U h ), β81 2 β82 (y) := .5 · (−ah (y) · S h · (I g + I h ) − au (y) · S h · (U g + U h )) · (rh (y) · I h − eh (y) · U h ), 2 β83 (y) := .5 · (−ag (y) · S h · (I g + I h ) − bg (y) · E g ) · (rh (y) · I h − eh (y) · U h ), 2 (y) := .5 · (ah (y) · S h · (I g + I h ) + au (y) · S h · (U g + U h ) − bh (y) · E h ) · (rh (y) · I h β84

−eh (y) · U h ), 2 β85 (y) := .5 · (bg (y) · E g − cg (y) · I g − rg (y) · I g ) · (rh (y) · I h − eh (y) · U h ), 2 β86 (y) := .5 · (bh (y) · E h − ch (y) · I h − rh (y) · I h ) · (rh (y) · I h − eh (y) · U h ), 2 β87 (y) := .5 · (rg (y) · I g − eg (y) · U g ) · (rh (y) · I h − eh (y) · U h ), 2 (y) := .5 · (rh (y) · I h − eh (y) · U h )2 . β88

156

7 Epidemic Models in Random Media

7.6 Concluding remarks In the model considered in this chapter, which concerns the SARS epidemic in Greater Toronto Area, we supposed that it has the pre-quarantine model [176]. Compare with another Model II-intra-quarantine. The continuous-time asymptotic results of this model are presented in [168]. The state space (7.9) of the merged switching process can be considered in a more general scheme, as countable or general measurable state space, but in any case, in order to be useful, it has to be much simpler than the initial one. The split and merging scheme can also be considered into several levels, particularly, in a double merging scheme, see, e.g., [90]. Moreover, the same conclusion can be obtained when the vector-valued function .g is replaced by the perturbed one ε ε 3/2 g (y, z) vanishes as .ε → 0. A non.g (y, z) = g (y, z)+εg1 (y, z) since the term .ε 1 ergodic split and merging scheme can also be considered here, see, e.g., Sect. 2.6.1. Of course, a particular case of switching process, which is simpler than the semiMarkov chain, is the Markov chain case. Finally, it is worth noticing that the discrete closed form formulae provided here are adapted for a direct computer calculus of the evolution of the different subpopulations as a function of time, once the statistical estimation of the coefficients is done from observed data. For further reading on this chapter the following references are of interest [117, 152, 155, 165–167].

Chapter 8

Optimal Stopping of Geometric Markov Renewal Chains and Pricing

8.1 Introduction This chapter is devoted to geometric Markov renewal chain (GMRC) and to discretetime semi-Markov-modulated .(B, S)-security markets and their properties. Optimal stopping rules for these models are investigated, and pricing formulas for European and American options are presented. We study discrete-time Markov-modulated .(B, S)-security market that consists of a bond B as a riskless asset, a stock S as a risky asset, and Markov renewal process that describes changes in the stock prices. The model for stock price we call it geometric Markov renewal process in discrete time by analogy with the geometric compound Poisson process introduced by Aase [1]. Discrete-time .(B, S)-securities market with the dynamic of bond price (riskless asset) .Bn = Bn−1 (1 + r), .r > 0, .B0 > 0, and with the dynamic of stock price (risky asset) .Sn = Sn−1 (1 + ρn ), .S0 > 0, where .ρn are i.i.d. random variables such that .ρn = b with probability p and .ρn = a with probability .q = 1 − p, .−1 < a < r < b, was proposed by Cox, Ross, and Rubinstein in 1976 [32]. Cox, Ross, and Rubinstein introduced this binomial model and derived the seminal Black–Scholes pricing formula for European call option. Our model is a generalization of Cox, Ross and Rubinstein [32] and Aase models [1]. Moreover, our model for stock price as the geometric Markov renewal process is an example of discrete -time semi-Markov random evolution [91, 157]. We study optimal stopping problems for geometric Markov renewal processes, and, based on these results, we find American option pricing formula for the discrete-time Markov-modulated .(B, S)-security markets. Also, we present European call option pricing formula for this model. American options can be exercised at any time the holder wishes to do so before its expiry date. European options can be exercised at the expiry date. Since American options give the right to its holder to exercise the option at any time before the option expires, traders are more attracted by this option because of this privilege. It therefore becomes © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0_8

157

158

8 Optimal Stopping of Geometric Markov Renewal Chains and Pricing

important to have knowledge on what is the best time to exercise American options in order to maximize the payoff. That is why this leads us to the study of American options as optimal stopping problems. American options pricing has been studied in many papers and books. See, for example, [17, 140] for continuous-time and [130, 135, 140–142] discrete-time settings, respectively. Optimal stopping problems have been studied in many papers and books as well, see, e.g., [28] and [141] that are the most close to our study. Averaging, merging, and double merging of geometric Markov renewal processes in series scheme and their applications in finance were studied in continuous time in [159]. Diffusion approximation of geometric Markov renewal processes in series scheme with merging and double merging approaches and option pricing formulas for the limit processes were studied in [160]. Normal deviations of the geometric Markov renewal processes in series scheme for ergodic averaging and double averaging schemes are derived in [161]. Poisson averaging scheme for the geometric Markov renewal processes is introduced in [161] as well. An analogue of Black–Scholes formula for diffusion Markov-modulated .(B, S)-security market and diffusion Markov-modulated .(B, S)-security market with jumps have been obtained in [169]. We mention that random evolution in general and discrete-time random evolutions in particular have many applications, including finance, biology, queuing, reliability, etc. Merging of random evolutions and their applications have been studied in [90, 91, 157]. Applications of discrete-time random evolutions in reliability and DNA analysis and their applications to difference equations and additive functionals have been considered in [12] and [101], respectively. This chapter is organized as follows. The geometric Markov renewal process (GMRP), discrete-time Markov-modulated .(B, S)-security market, and statement of the problem are presented in Sect. 8.2. In Sect. 8.3, we describe the GMRP as discrete-time jump semi-Markov random evolution. In Sect. 8.4, we study their martingale properties. Optimal stopping rules for GMRP are investigated in Sect. 8.5. Martingale properties of discount stock price and discount capital are considered in Sect. 8.6. American option pricing formulae for discrete-time Markovmodulated .(B, S)-security market are presented in Sect. 8.7. Here, we also consider the limit case when maturity goes to infinity (perpetual option). European option pricing formulae for discrete-time Markov-modulated .(B, S)-security market is derived in Sect. 8.8. Section 8.9 contains the proofs of two Lemmas related to the proof of our main optimal stopping problem result in Sect. 8.4 (Theorem 8.1). Some concluding remarks are given in Sect. 8.10.

8.2 GMRC and Embedded Markov-Modulated (B, S)-Security Markets

159

8.2 GMRC and Embedded Markov-Modulated (B, S)-Security Markets 8.2.1 Definition of the GMRC Let us consider a semi-Markov chain .(zk , k ∈ N) with state space a measurable space .(E, E ) and EMC .(xk , k ∈ N), with transition probability kernel .P (x, A) (see Chap. 2), where .x ∈ E, A ∈ E . Let .(τn , k ∈ N) be a sequence of jump times such that P(τn+1 − τn = k | xn = x) = fx (k),

.

where .x ∈ E, k ∈ N. Let us set θn+1 := τn+1 − τn ,

.

τn =

n 

.

θ ,

=1

and let νk := max{n : τn ≤ k}

.

be the counting process of jumps. Denote by q the semi-Markov kernel, i.e., q(x, B, k) = P (x, B)fx (k), and by .Q(x, B, k) the cumulative semi-Markov kernel, for .x ∈ E, .B ∈ E , and .k ∈ N. Denote also by .Fx the cumulative sojourn time distribution. We have (see Chap. 2)

.

Q(x, B, k) =

k 

.

q(x, B, s),

Fx (k) =

s=0

k 

fx (s).

s=0

Let us also consider the corresponding Markov renewal process .(xn , θn , n ∈ N) on the state space .E × N, and the semi-Markov chain .zk := xνk . Let .ρ(x) be a bounded continuous function on E such that .ρ(x) > −1. Define the following random functional on Markov renewal process .(xn , θn ) Sk := S0

νk 

.

(1 + ρ(xs )),

(8.1)

s=1

where .S0 > 0 is the initial value. This process is said to be a Geometric Markov Renewal Chain (GMRC or GMRP). We call it by analogy with the geometric

160

8 Optimal Stopping of Geometric Markov Renewal Chains and Pricing

compound Poisson process, in continuous time, defined as follows: S(t) = S0

N (t) 

(1 + Yk ),

.

k=1

where .t ∈ R+ , .S0 > 0; .N(t) is a standard Poisson process, and .(Yk , k ∈ IN) is an i.i.d. random sequence, which is a trading model in many financial applications as a pure jump model [162]. For continuous-time GMRC, see [162]. The GMRC in (8.1) will be our main model in further analysis in this chapter.

8.2.2 Statement of the Problem: Optimal Stopping Rule By embedded Markov-modulated .(B, S)-security market, we mean the market with riskless asset (bond) .Bn and risky asset (stock) .Sn that are defined in the following way:  .

Bn = B0 (1 + r)n ,  Sn = S0 ns=1 (1 + ρ(xs )),

(8.2)

where .r > 0 is an interest rate, .B0 > 0, .S0 > 0,, and functions .ρ(x) and .xn are defined in Sect. 8.2.1. Let .Fn := σ (xk ; 0 ≤ k ≤ n) be the .σ -algebra generated by the embedded Markov chain .(xn ). Denote by .g(s, x) a family of .B+ × E -measurable functions with values in .(−∞, +∞]. Let .τ be a stopping time, with respect to .Fn , such that .τ ≡ τ (ω) ≤ N, for .N < ∞, g(Sτ , xτ ) ≡ g(Sτ (ω) (ω), xτ (ω) (ω))

.

the random variable that is a gain in the state .(Sτ , xτ ) under censuring of observation at the moment .τ. The expectation .Es,x g(Sτ , xτ ) is a mean gain under the initial condition .{S0 = s, x0 = x}. Put CN (s, x) := sup Es,x g(Sτ , xτ ).

.

τ

The value .CN (s, x) is a price. The moment .τN∗ such that Es,x g(SτN∗ , xτN∗ ) = CN (s, x)

.

is called an optimal stopping time.

(8.3)

8 GMRP as Jump Discrete-Time Semi-Markov Random Evolution

161

Now, we are going to solve the following problems: (i) The structure of the price .CN (s, x) (ii) How to find .CN (s, x) (iii) How to find .τN∗

8.3 GMRP as Jump Discrete-Time Semi-Markov Random Evolution Let .C0 (R+ ) be the space of continuous function on .R+ , vanishing on the infinity, and let us define a family of contraction operators .D(x), on .C0 (R+ ), as follows: D(x)f (s) := f (s(1 + ρ(x))).

.

(8.4)

Then, by definition, jump random evolution is defined as the following product: Φk :=

νk 

.

D(x ).

=1

That is why, from (8.4), we obtain Φk f (s) =

νk 

.

D(x )f (s) = f (s

=1

νk 

(1 + ρ(x ))) = f (Sk ),

(8.5)

=1

where .Sk is defined in (8.1) with .S0 = s. Let us define the expectation of jump random evolution .Φk in (8.5): u(k, x, s) := Ex [Φk f (s, zk )].

.

Then function .u(k, x, s) satisfies the following Markov renewal equation: u(k, x, s) = F¯x (k)f (s, x) +

k  

.

Q(x, dy, )D(y)u(k − , y, s),

=0 E

where .F¯x (k) = 1 − Fx (k), and .f (s, x) is a bounded and continuous function on .R+ × E. Taking into account the above representations, we obtain that the function w(k, x, s) := Ex,s [f (Sk , zk )]

.

162

8 GMRP as Jump Discrete-Time Semi-Markov Random Evolution

satisfies the following Markov renewal equation: w(k, x, s) = F¯x (k)f (s, x) +

k  

Q(x, dy, )w(k − , y, s(1 + ρ(y))).

.

=0 E

This equation is a main tool in the investigation of limit distributions of the functional ν[tT ]



ST (t) = S0

.

(1 + T −1 ρ(xn )),

n=1

where .[a] denotes the integer part of the real nonnegative number a. It is one of the methods for obtaining all the limits for .ST (t) as .T → ∞. The second method is based on the martingale representation.

8.4 Martingale Properties of GMRC In this section, we consider the filtration .Fk := σ (z ; 0 ≤  ≤ k), .k ≥ 1. Let we have Sk = S0

νk 

.

(1 + ρ(x )).

(8.6)

=1

Let us define for all .k ∈ [0, T ], with .T ∈ N: Lk := L0

νk 

.

h(x ),

EL0 = 1,

=1

where .h(x) is a bounded continuous function, such that:   . h(y)P (x, dy) = 1, and h(y)P (x, dy)ρ(y) = 0. E

(8.7)

E

If .ELT = 1, then process .Sk in (8.6) is an .(Fk , P ∗ )-martingale, where measure .P ∗ is defined as follows: .

dP ∗ = LT . dP

8.5 Optimal Stopping Rules for GMRC

163

Another possibility in the discrete case is to consider the calendar time, and define the process .Sk as follows: Sk = S0

k 

.

(1 + ρ(zk )).

s=1

 Let .Lk := L0 ks=1 h(zs ), and .EL0 = 1, where .h(x) is defined in (8.7). If .ELN = ∗ 1, then .Sk is an .(Fk , P ∗ )-martingale, where . dP dP = LN .

8.5 Optimal Stopping Rules for GMRC Define the operator T by T g(s, x) := E[g(S1 , x1 ) | S0 = s, z0 = x] = Es,x g(s(1 + ρ(x1 )), x1 ),

.

(8.8)

and let Qg(s, x) := max{g(s, x); T g(s, x)}.

(8.9)

.

Let also Cn (s, x) :=

.

sup

τ ∈Mg (n)

Es,x g(Sτ , xτ ),

(8.10)

where Mg (n) := {τ : τ (ω) ≤ n; n < +∞; Es,x g− (Sτ , xτ ) < ∞, x ∈ E},

.

(8.11)

and g− (s, x) := − min{g(s, x), 0}.

.

From the definition of Q in (8.9), it follows that Qn g(s, x) = max{Qn−1 g(s, x); T Qn−1 g(s, x)},

.

n = 1, 2, . . . ,

where .Q0 g(s, x) = g(s, x). Also, Qn g(s, x) = max{g(s, x); T Qn−1 g(s, x)}.

.

(8.12)

164

8 GMRP as Jump Discrete-Time Semi-Markov Random Evolution

Theorem 8.1 Let .g(s, x) be such that .Es,x g− (S1 , x1 ) < ∞. Then: 1. Cn (s, x) = Qn g(s, x),

.

n ∈ N.

(8.13)

2. Cn (s, x) = max{g(s, x); T Cn−1 (s, x)},

.

(8.14)

where .C0 (s, x) = g(s, x). 3. The moment (stopping time) τn∗ := min{0 ≤ m ≤ n : Cn−m (Sm , xm ) = g(Sm , xm )}

.

(8.15)

is an optimal stopping time and Cn (s, x) = Es,x g(Sτn∗ , xτn∗ ).

.

(8.16)

Actually, for .n = 2, we obtain Q2 g(s, x) = max{Qg(s, x); T Qg(s, x)}

.

= max{max{g(s, x); T g(s, x)}; T {max{g(s, x); T g(s, x)}} = max{g(s, x); T {max{g(s, x); T g(s, x)}} = max{g(s, x); T Qg(s, x)}. If .n = 2, the formula (8.15) is true. To prove (8.15) for all n, we use the method of mathematical induction. The proof of Theorem 8.1 is based on the following two lemmas. Lemma 8.1 For every .τ ∈ Mg (n) : Es,x g(Sτ , xτ ) ≤ Qn g(s, x),

.

x ∈ E,

(8.17)

and, hence, Cn (s, x) ≤ Qn g(s, x).

(8.18)

Qn g(s, x) = Es,x g(Sσn , xσn ),

(8.19)

.

Lemma 8.2 For all .n ∈ N, .

8.5 Optimal Stopping Rules for GMRC

165

where σn := min{0 ≤ k ≤ n : Qn−k g(Sk , xk ) = g(Sk , xk )}.

.

(8.20)

The proofs of these lemmas will be done in Sect. 8.9. Let us define now the following sets: DnN := {(s, x) : CN −n (s, x) = g(s, x)},

.

0 ≤ n ≤ N.

(8.21)

Stopping time .τN∗ (optimal) in (8.3) is described in terms of “stopping sets” .DnN in (8.21): τN∗ = min{0 ≤ n ≤ N : (Sk , xk ) ∈ DnN }

.

(8.22)

that follows from Theorem 8.1. Namely, if .(s, x) ∈ D0N (.(s, x) := (S0 , x0 )), / D0N , then we continue our then optimal rule orders instant stopping. If .(s, x) ∈ observation, and, depending on .(S1 , x1 ), either stop our observation (if (.S1 , x1 ) ∈ D1N ) or continue our next observation (if .(S1 , x1 ) ∈ / D1N ), and so on. Obviously, that observation process will stop at the moment N as .DNN = N × E. Let us consider the “set of continuation of observations”: FnN = N × E \ DnN ;

.

0 ≤ n ≤ N.

Obviously, .FNN = Ø and FNN−1 = {(s, x) : C1 (s, x) > g(s, x)}

.

= {(s, x) : Qg(s, x) > g(s, x)} = {(s, x) : T g(s, x) > g(s, x)} = {(s, x) : (T − I )g(s, x) > 0}, where I is an identity operator. As .g(s, x) ≤ C1 (s, x) ≤ . . . ≤ CN (s, x), then .FnN satisfies the following relation: Ø = FNN ⊆ FNN−1 ⊆ . . . ⊆ F0N .

.

In particular, F0N = {(s, x) : CN (s, x) > g(s, x)} ⊇ {(s, x); (T − I )g(s, x) > 0}.

.

It is worth noticing that in order to find the stopping time .τN∗ it is necessary to know the prices .Cn (s, x), for all .n : 0 ≤ n ≤ N. In this way, to solve a problem on optimal stopping in the following class: .M (N ) := {τ : τ ≤ N ; N < +∞}, it is necessary to solve the stopping problems successively in the following classes:

166

8 GMRP as Jump Discrete-Time Semi-Markov Random Evolution

M (1), . . . , M (N − 1). Respectively, the prices are founded with the help of iterations of operator .Q :

.

C1 (s, x) = Qg(s, x); . . . , CN −1 (s, x) = QN −1 g(s, x),

.

or Cn (s, x) = max{g(s, x); T Cn−1 (s, x)}.

.

8.6 Martingale Properties of Discount Price and Discount Capital Let us define a discounted stock price: Dn :=

.

Sn , Bn

(8.23)

where .Sn and .Bn are defined in (8.2). Then   1 + ρ(xn ) | Fn−1 Ex [Dn | Fn−1 ] = Ex Dn−1 1+r 1 + ρ(xn ) ] = Dn−1 , = Dn−1 Exn−1 [ 1+r

.

if Exn−1

.

1 + ρ(x )

n = 1, 1+r

or  P (x, dy)ρ(y) = r,

.

(8.24)

E

where .P (x, dy) is the transition probability kernel of the embedded Markov chain (xn , n ∈ N). In this way, the sequence .Dn is an .Fn -martingale if (8.24) is fulfilled. Let us define

.

Xn := βn Bn + γn Sn ,

.

(8.25)

with .(βn , γn ) being an investor’s portfolio (which is .Fn−1 -measurable), and (Bn , Sn ) being defined in (8.2).

.

8.6 Martingale Properties of Discount Price and Discount Capital

167

We suppose that our portfolio .(βn , γn ) is self-financing, i.e., .πn := (βn , γn ) ∈ SF : βn Bn + γn Sn = 0.

.

(8.26)

Then from (8.25) and (8.26), it follows that Xn = βn Bn + γn Sn = βn (rBn−1 ) + γn (ρ(xn )Sn ).

.

(8.27)

Put Mn :=

.

Xn Bn

(8.28)

for discounted capital. Then from (8.27), it follows that Xn−1 Xn − Bn Bn−1

Mn =

.

=

Xn − Xn−1 − rXn−1 Bn

=

γn Sn−1 (ρ(xn ) − r) . Bn

(8.29)

Let mn :=

.

n  (ρ(xk ) − r) k=1

and mn = (ρ(xn ) − r).

.

(8.30)

From (8.30), it follows that, for .n ≥ 1, M n = M0 +

.

n  γn Sk−1 k=1

Bk

mk .

(8.31)

Let .E∗ be an expectation by .P (x, A) as in (8.24). Then .mn in (8.30) is an .Fn martingale by .E∗ . Values .γk and .Sk−1 are .Fk−1 -measurable; hence, sequence .(Mn , Fn ) is also a martingale, where .Mn is defined in (8.28) (see also (8.31)). That is why E∗ MN = M0 .

.

(8.32)

168

8 GMRP as Jump Discrete-Time Semi-Markov Random Evolution

From (8.32) and (8.28), we obtain E∗ (1 + r)−N XN = x,

.

X0 = x.

(8.33)

A portfolio .πn := (βn , γn ) is a .(x, fN )-hedge, if for given .x > 0 and nonnegative function .fN (x) := fN (S0 , S1 , . . . , SN , x1 , . . . , xN ), .X0 = x, and we have XN ≥ fN (x).

.

(8.34)

If XN = fN (x),

.

then .πn is a minimal .(x, fN )-hedge. Let .Π (x, fN ) be a family of all .(x, fN )-hedges π ∈ SF . From (8.33) and (8.34), we obtain that if .πn ∈ Π (x, fN ), then

.

x ≤ E∗ (1 + r)−N fN (x).

.

(8.35)

If the hedge .πn is a minimal one, then x = E∗ (1 + r)−N fN (x),

.

which follows from (8.35) and (8.33). In this way, we obtain the following result. Lemma 8.3 Let on discrete .(B, S, X)-securities market the portfolio .πn = (βn , γn ) ∈ SF is .(x, fN )-hedge. Then .x ≥ E∗ (1 + r)−N fN (x). If .(x, fN )-hedge ∗ −N f (x), where .E∗ is an expectation by .πn is a minimal one, then .x = E (1 + r) N .P (x, A) in (8.24), .x ∈ E, .A ∈ E .

8.7 American Option Pricing Formulae for embedded Markov-modulated (B, S)-Security markets As .Mn = Xn /Bn is an .Fn -martingale, and .N < +∞, then for any stopping time τ , such that .τ ≤ N, we have .E∗ Mτ = M0 , namely,

.

E∗ (1 + r)−τ Xτ = X0 .

.

(8.36)

Let us suppose that portfolio .π is .(x, f, N)-hedge, namely, .X0 = x and .Xn ≥ fn (S0 , S1 , . . . , Sn , x1 , . . . , xn ) for any .0 ≤ n ≤ N. Then from (8.36), we obtain that x ≥ sup E∗ (1 + r)−τ fτ (x).

.

0≤τ ≤N

8.7 American Option Pricing Formulae

169

If the .(x, f, N)-hedge .πn is a minimal one (that is, there exists a stopping time σ such that for all .ω ∈ Ω, .Xσ = fσ (x)), then .x = X0 = E∗ (1 + r)−σ Xσ = E∗ (1 + r)−σ fσ , and hence,

.

x = sup E∗ (1 + r)−τ fτ (x).

.

0≤τ ≤N

(8.37)

∗ of American From here, we obtain the following result: the rational price .CN option with maturity date N and system of nonnegative cost functions .f (x) = (fn (x), 0 ≤ n ≤ N), for discrete .(B, S, X)-securities market, is defined by the following formula: ∗ CN = sup E∗ (1 + r)−τ fτ (x).

.

0≤τ ≤N

(8.38)

A stopping time .τ ∗ is rational if and only if ∗

E∗ (1 + r)−τ fτ ∗ (x) = sup E∗ (1 + r)−τ fτ (x).

.

0≤τ ≤N

∗ and rational stopping time In this way, the problem of finding a rational price .CN is solved by solving of one problem: optimal stopping of .

sup E∗ (1 + r)−τ fτ (x). τ

Let fn := β n g(Sn , xn ),

.

(8.39)

where .0 < β ≤ 1. We will solve this problem in the last case of (8.39). Let us put Vn (s, x) := sup E∗s,x (1 + r)−τ β τ g(Sτ , xτ ),

.

0≤τ ≤N

(8.40)

and T g(s, x) := E∗s,x g(S1 , x1 ).

.

Then T g(s, x) =

.

E∗s,x g(s(1 + ρ(x1 )), x1 )

 =

P (x, dy)g(s(1 + ρ(y)), y). E

(8.41)

170

8 GMRP as Jump Discrete-Time Semi-Markov Random Evolution

Put Qβ g(s, x) := max{g(s, x); (1 + r)−1 βT g(s, x)}.

.

Then, with respect to the previous results (see Sect. 8.2, Theorem 8.1), we obtain: 1. The function .Vn (s, x) in (8.40) is represented in the explicit form: Vn (s, x) = Qnβ g(s, x),

.

where .Qnβ is an .n − th power of operator .Qβ . 2. Vn (s, x) = max{g(s, x); (1 + r)−1 βT Vn−1 (s, x)}.

.

V0 (s, x) = g(s, x).

.

3. The stopping time τn := min{0 ≤ m ≤ n : Vn−m (Sm , xm ) = g(Sm , xm )}

.

(8.42)

is an optimal one E∗s,x ((1 + r)−1 β)τn g(Sτn , xτn ) = Vn (s, x).

.

(8.43)

Let Dm := {(s, x) : Vm (s, x) = g(s, x)},

.

Fm := {(s, x) : Vm (s, x) > g(s, x)} = N × E \ Dm .

.

(8.44) (8.45)

We note that Fn ⊇ Fn−1 ⊇ . . . ⊇ F0 = Ø;

.

Dn ⊆ Dn−1 ⊆ . . . ⊆ D0 = N × E.

.

From (8.42) and (8.43), it follows that the stopping time τm = min{0 ≤ m ≤ n : (Sm , xm ) ∈ Dn−m }

.

is an optimal one. Sets .Dn , Dn−1 , . . . , D0 are “stopping sets”, and sets Fn , Fn−1 , . . . , F0 are “sets of continuation of observations”.

.

8.7 American Option Pricing Formulae

171

From (8.44)–(8.45), it obviously follows that D0 = N × E,

.

D1 = {(s, x) : Qβ g(s, x) = g(s, x)},

.

D2 = {(s, x) : Q2β g(s, x) = g(s, x)},

.

and so on. Limit Case: .N → ∞ As .V0 (s, x) ≤ V1 (s, x) ≤ . . . , then there exists a limit .limn→+∞ Vn (s, x) = V (s, x). This function has the following properties: 1. V (s, x) = sup E∗s,x ((1 + r)−1 β)τ g(Sτ , xτ ).

.

(8.46)

τ

2. V (s, x) = max{g(s, x), (1 + r)−1 βT V (s, x)}.

.

3. .V (s, x) is the least of functions .u(s, x) ≥ 0 such that: u(s, x) ≥ g(s, x)

u(s, x) ≥ ((1 + r)−1 βT u(s, x).

and

.

4. The stopping time τ∞ := inf{n : V (Sn , xn ) = g(Sn , xn )}

.

is an optimal one, that is, V (s, x) = E∗s,x ((1 + r)−1 β)τ∞ g(S∞ , x∞ ).

.

If .C ∗ (s, x) is a set of continuation of observations, then .V (s, x) > g(s, x), and from (8.46), it follows that: V (s, x) = (1 + r)−1 βT V (s, x),

.

namely, −1

V (s, x) = (1 + r)

.

 P (x, dy)V (s(1 + ρ(y)), y).

β E

172

8 GMRP as Jump Discrete-Time Semi-Markov Random Evolution

8.8 European Option Pricing Formula for Embedded Markov-Modulated (B, S)-Security Markets Let us consider European call option, where the dynamic of stock price is described by embedded Markov-modulated .(B, S)-security market:  .

Bn = B0 (1 + r)n ,  Sn = S0 nk=1 (1 + ρ(xk )),

where .1 ≤ n ≤ N and N is a maturity date. Let f (SN ) = (SN − K)+ ,

.

where K is a strike price. From Sect. 8.6, it follows that optimal (or rational) price of European call option is equal to CN (x) = E∗ (1 + r)−N f (SN ) = E∗ (1 + r)−N (SN − K)+ ,

.

where .E∗ is an expectation by .P (x, dy) such that (8.47), we obtain .

 .

E

(8.47)

P (x, dy)ρ(y) = r. From

CN (x) = E∗ (1 + r)−N (SN − K)+    + = (1 + r)−N E . . . E (S0 N i=1 (1 + ρ(xi )) − K) P (yi−1 , dyi ).

Let us consider the geometric Markov renewal chain: Sk = S0

.

νk 

(1 + ρ(xj )),

j =1

where .(xn , θn , n ∈ N) is a Markov renewal chain, .xn ∈ E, .θn ∈ N, .νk := max{n : τn ≤ k} is the counting process of jumps, .τn := nk=1 θk , .θ0 = 0. Let f (SN ) = (SN − K)+ ,

.

where .SN = S0

νN

k=1 (1 + ρ(xk )).

8.9 Proof of Theorems

173

The price of a European call option is CN (x) = E∗ [f (SN )(1 + r)−N ] = E∗ [(1 + r)−N E∗ [f (SN ) | νN ]]   k ∞   = P(νN = k) . . . (S0 (1 + ρ(yi )) − K)+ P (yi−1 , dyi ).

.

E

k=0

E

i=1

(8.48)

8.9 Proof of Theorems In this last section, we will prove Lemmas 8.1 and 8.2 and Theorem 8.1. Proof of Lemma 8.1 If .n = 0, then (8.16) is obviously follows. Let now .τ ∈ Mg (n), .n > 0. Put .B := {ω : τ (ω) = n}. Then .B = Ω \ n−1 i=0 {τ = i} ∈ Fn−1 , and  .Es,x g(Sτ , xτ ) = Es,x 1B 1B g(Sτ , xτ ) ¯ g(Sτ , xτ ) + τ

= Es,x 1B¯ g(Sτ ∧(n−1) , xτ ∧(n−1) ) + Es,x 1B g(Sn , xn ) = Es,x 1B¯ g(Sτ ∧(n−1) , xτ ∧(n−1) ) + Es,x {1B Es,x {g(Sn , xn )|Fn−1 }} = Es,x 1B¯ g(Sτ ∧(n−1) , xτ ∧(n−1) ) + Es,x 1B ESn−1 ,xn−1 g(S1 , x1 ) = Es,x 1B¯ g(Sτ ∧(n−1) , xτ ∧(n−1) ) + Es,x 1B ESτ ∧(n−1) ,xτ ∧(n−1) g(S1 , x1 ) ≤ Es,x max[g(Sτ ∧(n−1) , xτ ∧(n−1) ), ESτ ∧(n−1) ,xτ ∧(n−1) g(S1 , x1 )] = Es,x Qg(Sτ ∧(n−1) , xτ ∧(n−1) ). So Es,x g(Sτ , xτ ) ≤ Es,x Qg(Sτ ∧(n−1) , xτ ∧(n−1) ).

.

(8.49)

Finally, we obtain from (8.49): Es,x g(Sτ , xτ ) ≤ Es,x Qg(Sτ ∧(n−1) , xτ ∧(n−1) )

.

≤ Es,x Q2 g(Sτ ∧(n−2) , xτ ∧(n−2) ) ··· ≤ Es,x Qn g(Sτ ∧0 , xτ ∧0 ) = Qn g(s, x), which proves (8.16) and (8.18), and Lemma 8.1 is proved.



.

174

8 GMRP as Jump Discrete-Time Semi-Markov Random Evolution

Proof of Lemma 8.2 The proof will be done by the method of mathematical induction. If .n = 0, then statement of Lemma 8.2 is obvious. Let the equality (8.19) is true for some .n ≥ 0. Let us show that (8.19) is also true for .(n + 1). Let us fix the point .(s, x) ∈ N × E. Then, if .P(σn+1 = 0) = 1, then by (8.20) we get P(Qn+1 g(S0 , x0 ) = g(S0 , x0 )) = 1,

.

and, hence, Qn+1 g(s, x) = g(s, x) = Es,x g(Sσn+1 , xσn+1 ).

.

Let now .Ps,x (σn+1 = 0) < 1. Then, since .{σn+1 = 0} ∈ F0 , by the “0-1” law Ps,x (σn+1 = 0) = 0, and, hence, .Ps,x (σn+1 ≥ 1) = 1. Let show that in this case .σn+1 = 1 + θ1 σn , where .θ1 σn := σn (θ1 ω) := σn+1 (ω). Really, .

θ1 σn (ω) = θ1 min{0 ≤ k ≤ n : Qn−k g(Sk (ω), xk (ω)) = g(Sk (ω), xk (ω))}

.

= min{0 ≤ k ≤ n : Qn−k g(Sk (θ1 ω), xk (θ1 ω)) = g(Sk (θ1 ω), xk (θ1 ω))} = min{0 ≤ k ≤ n : Qn−k g(Sk+1 (ω), xk+1 (ω)) = g(Sk+1 (ω), xk+1 (ω))} = min{0 ≤ k ≤ n : Qn+1−(k+1) g(Sk+1 (ω), xk+1 (ω)) = g(Sk+1 (ω), xk+1 (ω))}, where .θn is the shift operator, see Appendix A. Hence, 1 + θ1 σn (ω) = min{1 ≤ k + 1 ≤ n + 1 : Qn+1−(k+1) g(Sk+1 , xk+1 )

.

= g(Sk+1 , xk+1 )} = min{1 ≤ l ≤ n + 1 : Qn+1−l g(Sl , xl ) = g(Sl , xl )} = σn+1 (ω), where the last equality follows from definition of .σn+1 in (8.20) and preposition P(σn+1 ≥ 1) = 1. Here, .Qn+1 g(s, x) > g(s, x). From here, and induction preposition and from (8.15), we obtain

.

Qn+1 g(s, x) = max{g(s, x); Es,x Qn g(S1 , x1 )}

.

= Es,x Es,x g(Sσn , xσn ) = Es,x θ1 g(Sσn , xσn ) = Es,x g(S1+θ1 σ1 , x1+θ1 σn ) = Es,x g(Sσn+1 , xσn+1 ), which complete the proof of Lemma 8.2.



.

8.10 Concluding Remarks

175

Proof of Theorem 8.1 From (8.18) and (8.19), it follows that Cn (s, x) ≤ Qn g(s, x) = Es,x g(Sσn , xσn ).

.

Obviously, Cn (s, x) ≥ Es,x g(Sσn , xσn ),

.

as .Cn (s, x) is a price, namely, .sup Es,x g. In this way, for all .n = 0, 1, 2, . . ., Cn (s, x) = Qn g(s, x) = Es,x g(Sσn , xσn ),

.

and, hence, the stopping time .σn (.= τn∗ ) is an optimal one, where .σn is defined in (8.20). Statement 2) (see (8.14)) follows from 1), (8.6), and (8.15). Theorem 8.1 is . proved.

8.10 Concluding Remarks We presented geometric Markov renewal processes and based on the embedded Markov-modulated .(B, S)-security markets and studied their properties. Optimal stopping rules for these models have been investigated, and pricing formulas for European and American options have been presented. A valuable extension of the present work could be associated with optimal stopping rules for discrete-time jump semi-Markov random evolutions and implementations of obtained results for calendar time .k ∈ N semi-Markov-modulated .(B, S)-security markets instead of jump times n.

Appendix A

Markov Chains

Markov chains are particular cases of semi-Markov chains, and moreover they are present as the embedded Markov chains in any semi-Markov chain. We present here the basic definition and some results from Markov chains theory with general state space. For Markov chains, see, e.g., Gihman-Skorokhod [54], Kartashov [81], Nummelin [120], and Revuz [136].

A.1 Transition Function Let .E = Ø be a set and .E a countably generated .σ -algebra of subsets of E. Definition A.1 A function .P : E × E → [0, 1] is called a transition function (or a kernel or a Markov kernel) on .(E, E ) if the following conditions are fulfilled: 1. .P (x, · ) is a measure on .E for each .x ∈ E. 2. .P ( · , B) is an .E -measurable function for each .B ∈ E . The transition function is called substochastic or stochastic as .P (x, E) ≤ 1 for some x ∈ E or .P (x, E) = 1 for each .x ∈ E, respectively. The composition of any two transition probabilities, say P and Q, on .(E, E ), is defined as  .P Q(x, A) = P (x, dy) Q(y, A), x ∈ E, A ∈ E .

.

E

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0

177

178

A Markov Chains

The transition functions in n-steps are defined by the relations .

P 0 (x, B) = δx (B);

(the Dirac measure at x ∈ E)

P (x, B) = P (x, B);   n+1 n (x, B) = P (x, dy) P (y, B) = P n (x, dy) P (y, B), P 1

E

(A.1) n ≥ 1,

E

for .x ∈ E, B ∈ E . It is worth noticing here that when .x ∈ E is fixed and B is any element of .E , we write the Dirac measure .δx (B), and when B is fixed and x is any element of E, then we write the indicator function .1B (x). Moreover, for a Borel measurable bounded function .f : E → R+ and a measure .μ on the measurable space .(E, E ), we define the transition operator P , related to the transition kernel .P (x, B), as follows:  • .Pf (x) = P (x, dy)f (y), x ∈ E. E   • .(μf )(A) = μ(f 1A ) = f (x)μ(dx) and (μP )(A) = μ(dx)P (x, A), A

x ∈ E,

.

E

A ∈ E.

Example A.1 If E is a finite or countable set and .E = P(E), then we set P (x, {y}) = p(x, y) for all .x, y ∈ E.

.

Let .(Ω, F , P) be a probability space and .P (x, B) be a transition probability on (E, E ).

.

Definition A.2 A stochastic process .X = (Xn , n ∈ N) defined on .(Ω, F , P) with values in .(E, E ) is called homogeneous (with respect to the time) Markov chain with the transition probabilities P if P(Xn+1 ∈ A | σ (Xk , k ≤ n)) = P(Xn+1 ∈ A | Xn ) = P (Xn , A),

.

P − a.s. (A.2)

for all .n ∈ N, A ∈ E . For a countable set E, the Markov process with state space .(E, P(E )) is defined in a similar way. A probability measure .μ, on .E , is called the initial distribution of the Markov chain X, if .X0 ∼ μ. If .μ is a probability with a single atom in .x ∈ E, that is, .μ(A) = δx (A), A ∈ E , then the corresponding probability on .F∞ is denoted by .Px , and, for any probability measure .μ, we can write  .Pμ (·) = μ(dx)Px (·). (A.3) E

A.1 Transition Function

179

The corresponding expectation is denoted by .Eμ . Let .Fn = σ (Xk , k ≤ n), n ∈ N, and .F∞ = σ (Xk ; k ∈ N). According to the properties of the conditional probabilities, we get from (A.2) that P(Xn+1 ∈ A1 , Xn+2 ∈ A2 , · · · , Xn+m ∈ Am | Fn )     = P ( · , dx1 ) P (x1 , dx2 ) . . . P (xm−1 , dxm ) ◦ Xn ,

.

A1

A2

P − a.s.

Am

(A.4) for all .m, n ∈ N, .A1 , A2 , · · · , Am ∈ E . Particularly, P(Xn+m ∈ A | Fn ) = P(Xn+m ∈ A | Xn ) = P m (Xn , A),

.

Px − a.s.

(A.5)

for all .m, n ∈ N, .A ∈ E . So, for each probability .μ on .E , there is a probability .Pμ on .F∞ , such that .

Pμ (X0 ∈ A0 , X1 ∈ A1 , · · · , Xn ∈ An )    = μ(dx0 ) P (x, dx1 ) . . . P (xn−1 , dxn ) A0

A1

(A.6)

An

for .n ∈ N, .A0 , A1 , · · · , An ∈ E . Now we shall consider the case of a substochastic (sub-Markov) kernel .P (x, B), .x ∈ E, B ∈ E (that is, .P (x, E) ≤ 1). Let .Δ be a point that not belongs to E and .EΔ = E ∪ {Δ} , EΔ = σ (E , {Δ}). The transition function P will be extend to .(EΔ , EΔ ) as follows: P (x, {Δ}) = 1 − P (x, E),

.

x ∈ E, and

P (Δ, {Δ}) = 1.

(A.7)

This extended function is a Markov kernel. Moreover, any probability .μ on .(E, E ) can be extended to .(EΔ , EΔ ) by setting .μ({Δ}) = 0. In this way, we obtain a Markov process with values in .(EΔ , EΔ ). The point .Δ is called “the cemetery” of the process. The time .ζ when the process enter .Δ gives its life time. Obviously, the random variable .ζ is a stopping time and it says that the process “dies” at the time .ζ . Definition A.3 The function .θ : E ∞ → E ∞ , such that, θ (x0 , x1 , · · · ) = (x1 , x2 , · · · ),

.

xk ∈ E,

(A.8)

is called a shift operator. Obviously, .θ is measurable with respect to the .σ -algebra E∞ = E × E × ....

.

180

A Markov Chains

For .m ∈ N, the operator .θm is defined by recurrence as θ0 = 1E ∞ ;

.

θm = θ ◦ θm−1 ,

m≥1

We have clearly that .Xn ◦ θm = Xn+m . Let .Y : Ω → E be measurable with respect to .F (= F∞ ). One can prove that there is a measurable function .h : E ∞ → E ∞ such that Y (ω) = h(X0 (ω), X1 (ω), · · · )

.

for any ω ∈ Ω.

Hence, we have (Y ◦ θm )(ω) = h(Xm (ω), Xm+1 (ω), · · · )

.

for any ω ∈ Ω

(A.9)

and Ex (Y ◦ θn | Fn ) = EXn (Y ),

.

Px − a.s.

(A.10)

on the set .{Xn = Δ}, for all .n ∈ N. The relation (A.10) is a consequence of (A.2) by using a usual class monotone argument (see, e.g., [16] or [136]). Like for a countable state space, the Markov property (A.2) implies the strong Markov property. More exactly, let .τ : Ω → R be a stopping time; then the shift .θτ is defined by   (Y ◦ θτ )(ω) = h Xτ (ω) (ω), Xτ (ω)+1 (ω), · · · ,

.

ω∈Ω

(A.11)

and the following proposition can be proved as in countable state space case. Proposition A.1 Let .(Xn , n ∈ N) be a Markov chain with values in .(E, E ) and with transition function .P (x, B), .x ∈ E, B ∈ E . Then, for any stopping time .τ : Ω → N, we have     Px Xτ +1 ∈ A | Fτ = P Xτ , A ,

.

Px − a.s.

(A.12)

for all .x ∈ E, A ∈ E . Moreover,   Ex Y ◦ θτ | Fτ = EXτ (Y ),

.

Px − a.s.

(A.13)

for any measurable function .Y : Ω → E. Example A.2 Let .(Xn , n ∈ N∗ ) be a sequence of i.i.d. random variables with the common law .λ, that is, λ(A) = P(Xn ∈ A),

.

n ∈ N∗ , A ∈ E .

A

Markov Chains

181

Moreover, let .X0 be another random variable, with law .μ, independent of .(Xn , n ∈ N∗ ). We define Zn = X0 +

n 

.

Xk ,

n ∈ N∗ ,

Z0 = X0 .

(A.14)

k=1

In case where .(E, E ) = (R, B), with .B the Borel subsets of .R, the stochastic process .(Zn ; n ∈ N) is called a random walk on .R. This real-valued process is a Markov chain whose transition function is P (x, A) = λ(A − x),

.

x ∈ R, A ∈ B,

(A.15)

where A − x := {y − x : y ∈ A}.

.

The probability .μ is the initial law or probability of the chain.

A.2 Irreducible Markov Chains Let .(Xn , n ∈ N) be a Markov chain with state space .(E, E ) and whose transition probability is .P (x, A). Definition A.4 The set .A ∈ E is said to be accessible from the state .x ∈ E (denoted by .x → A) if there exists .n ∈ N∗ such that .P n (x, A) > 0. In other words, .x → A if and only if .L(x, A) > 0, where .L(x, A) = Px (∪n∈N∗ (Xn ∈ A)). If .L(x, A) = 0, we write .x → A. Definition A.5 A non-empty set .F ∈ E is called closed if .P (x, F ) = 1 for any x ∈ F . A closed set that does not contain two disjoint closed subsets is called indecomposable.

.

Definition A.6 Let .ϕ be a .σ -finite measure on .(E, E ). The Markov chain .(Xn , n ∈ N) is called .ϕ-irreducible if .L(x, A) > 0 for any .x ∈ E provided that .ϕ(A) > 0. The chain is called irreducible if it is .ϕ-irreducible with respect to an arbitrary .σ finite measure with .ϕ(E) > 0.

182

A Markov Chains

Let us consider a Markov chain X with .π -irreducible transition kernel P on the state space .(E, E ), and suppose that there is a partition of the state space E of measurable sets in .E , say .C0 , .C1 ,. . . , .Cd−1 , H , with .d ≥ 1, such that, for .i = 0, 1, . . . , d − 1 and .x ∈ Ci , P (x, Cjc ) = 0,

.

j = i + 1(mod d).

for

and .π(H ) = 0. This sequence is called a d-cycle for P and the sets .C0 , .C1 ,. . . , .Cd−1 are called cyclic sets. The cyclic decomposition is unique up to equivalence. The integer d is called the period of P . If .d = 1, P (or the Markov chain X) is called aperiodic, and if .d > 1, P is called periodic with period d or d-periodic.

A.3 Recurrent Markov Chains Let us consider a sequence of random variables .X = (Xn , n ∈ N) defined on (Ω, F , P) with values in .(E, E ). n We  set .Fn = σ (Xk ; k ≤ n), .F = σ (Xk ; k ≥ n), .F∞ = σ (Xk ; k ∈ N) = σ n∈N Fn , and suppose .F∞ = F . Definition A.7 The .σ -algebra .F ∞ = F n is called the asymptotic .σ -algebra.

.

n∈N

This name is justified by the fact that any event .A ∈ F n is independent of the random variables .X1 , X2 , · · · , Xn for each .n ∈ N, but it is determined by an infinite number of random variables from the sequence X. The following events are some examples of asymptotic events in the case where .(E, E ) = (R, B):



.

 Xn converges ;

n∈N

.



  Xn converges ; n

n∈N



lim sup Xn < ∞ ; n→∞

X1 + X2 + · · · + Xn converges . lim sup n n→∞

We are interested in conditions which ensure that the asymptotic .σ -algebra is P-trivial (that is, .P(A) = 0 or 1 for any .A ∈ F ∞ ). For each .A ∈ E , we set

.

Q(x, A) = Px (Xn ∈ A, i. o.).

.

(A.16)

A

Markov Chains

183

The event .(Xn ∈ A, i. o.) can be still expressed as follows: ∞ ∞ 

(Xn ∈ A, i.o.) =

.

  (Xn ∈ A) = lim sup 1A (Xn ) = 1 n→∞

n=1 m=n

 =





1A (Xn ) = ∞ .

(A.17)

n∈N∗

For .A ∈ E , let .TA be the first entry into the set A, that is, TA = inf{n > 0 : Xn ∈ A}

.

with .inf Ø = ∞. The r.v. .TA is a stopping time since

.

{TA = n} =

n−1

(Xm ∈ Ac ) ∩ (Xn ∈ A) ∈ Fn .

m=1

It is not difficult to see that (Xn ∈ A, i. o.) =



.

(n + TA ◦ θn < ∞).

(A.18)

n∈N

Definition A.8 A set .A ∈ E is called recurrent (transient) if .Q(x, A) = 1 .(< 1) for all .x ∈ E. A set can be neither recurrent nor transient. However, if the .σ -algebra .F ∞ is .Pμ trivial for any probability .μ on .E , then each set .A ∈ E is either recurrent or transient. This is a consequence of the belongingness .(Xn ∈ A, i.o.) ∈ F ∞ . There are necessary and sufficient conditions in order to each set .A ∈ E to be either recurrent or transient. The following definition and proposition are important. Definition A.9 Let .ϕ be a .σ -finite measure on .E with .ϕ(E) > 0. The chain X is called .ϕ-recurrent or Harris recurrent, if any set .A ∈ E with .ϕ(A) > 0 is recurrent (cf. to Definition A.8) Obviously, a chain .ϕ-recurrent is .ϕ-irreducible (cf. to Definition A.6). Proposition A.2 Let X be a .ϕ-recurrent chain. Then each set .A ∈ E is either recurrent or transient.

184

A Markov Chains

A.4 Invariant Measures Definition A.10 A σ -finite measureπ on E is called invariant for the stochastic kernel P (x, A) if π P = π , that is, E π(dx)P (x, A) = π(A). The measure π is still called invariant for the Markov chain X whose transition function is P . If π is a probability, then it is called stationary probability. For a ϕ-recurrent chain, there is an invariant measure. More precisely, we have the following proposition (for proof, see, e.g., [121], p. 31). Proposition A.3 For a ϕ-recurrent Markov chain, there is a non-trivial σ -finite measure π such that: 1. π is invariant. 2. If π is another σ -finite and invariant measure, then π is multiple of π . 3. ϕ is absolute continuous with respect to π . Corollary A.1 Let π be a non-trivial σ -finite and invariant measure, on (E, E ). If the chain X is ϕ-recurrent, then, for any B ∈ E , the following assertions are equivalent: 1. π(B) > 0. 2. L(x, B) > 0 for any x ∈ E. 3. Q(x, B) = 1 for any x ∈ E. Proposition A.4 Let X be a ϕ-recurrent Markov chain and μ and nu be two probabilities on E . We have: 1. If X is aperiodic, then .

  lim (μ − ν) P n  = 0.

n→∞

In particular, if π is an invariant probability, then .

  lim μ P n − π  = 0.

n→∞

2. If X is periodic with the period d, then 1 . lim n→∞ d

  d    (nd+k)   = 0.  (μ − ν) P   k=1

Now we shall introduce a stronger condition than the ϕ-recurrence. This is necessary to study the existence of an invariant probability. Firstly, for A, B ∈ E , we consider the probability of the entrance of the process into A at the moment m before it enters B. More precisely, .B

P m (x, A) = Px (Xm ∈ A, Xi ∈ / B, Xi ∈ / A, 1 ≤ i < m).

A

Markov Chains

185

So, A P m (x, A) is the probability of the first entrance into A at the moment m and we have .A

P m (x, A) = Px (TA = m).

(A.19)

Definition A.11 The chain X = (Xn , n ∈ N) is called uniformly ϕ-recurrent if .

lim

n→∞

n 

AP

m

(x, A) = 1

(A.20)

m=1

uniformly with respect to x, whenever ϕ(A) > 0. Using (A.19), it is not difficult to see that the relation (A.20) is equivalent to each of the following relations: .

limn→∞ Px (TA ≤ n) = 1  n limn→∞ Px m=1 (Xm ∈ A) = 1.

(A.21)

For x ∈ E, A, B ∈ E , B ⊂ A, the probability PA (x, B) =

∞ 

.

AP

m

(x, B)

m=1

means the probability that the first visit paid by the process to the set A to be a visit paid to B. If Q(x, A) = 1 for any x ∈ A, then PA (x, ·) is a transition function on (A, EA ), where EA is the class of all subsets of A which belongs to E . Let T1 , T2 , · · · be a sequence of stopping times defined by recurrence as follows: T1 = TA ,

.

T2 = T1 + T1 ◦ θT1 , · · · , Tn = Tn−1 + TA ◦ θTn−1 , · · · .

They are successive moments of passage through the set A. The process X0 , XT1 , XT2 , · · · is a Markov chain with the transition function PA (x, B), x ∈ A, B ∈ EA . This chain is called the process on A. Proposition A.5 Let us consider a Markov chain X uniformly ϕ-recurrent (cf. to Definition A.11) and μ and ν two probabilities on E . We have: 1. If X is aperiodic, then there are two nonnegative constants a < ∞ and ρ < 1, such that .

  (μ − ν) P n  ≤ aρ n μ − ν .

(A.22)

186

A Markov Chains

2. If X is periodic with the period d > 1, there are two nonnegative constants a < ∞ and ρ < 1, such that  n   1   k (μ − ν) P  ≤ aρ n μ − ν . (A.23) .  n k=1

If the initial probability α of a Markov chain P is its stationary probability π , then the Markov chain is stationary. A set B ∈ E is called π -invariant if π(x ∈ B, P (x, B) < 1) = 0. π is an ergodic distribution if π(B) = 0 or π(B) = 1 for every π -invariant set B. Nonhomogeneous Markov chain. A nonhomogeneous Markov chain (Xn , n ≥ 0) with transition kernels Pn , n ≥ 0, then the Markov chain Yn = (n, Xn ) is a homogeneous Markov chain, with transition kernel Q defined as Q((n, x), C × D) = Pn (x, D)1C (n + 1), for C ⊂ N, D ∈ E .

A.5 Uniformly Ergodic Markov Chains Here we give some definitions, see Kartashov [81] and also [18, 40, 90–92]. Let us consider a complete linear normed space, that is, a Banach space .B and denote by .· its norm. Let A be a linear operator acting in .B, and define its domain .D(A), its range .R(A), and its null space .N(A), that is, • .D(A) := {f ∈ B : Af ∈ B}. • .R(A) := {g ∈ B : ∃f ∈ B, Af = g}. • .N (A) := {f ∈ D(A) : Af = 0}. The space .N (A) is said to be non-trivial if it contains at least one non-zero element, and then we have .dim N(A) ≥ 1. The operator A is said to be densely defined if .D(A) is dense in .B, that is, ∗ ∗ ∗ .D(A) = B. For a densely defined operator, its adjoint operator, .A : B −→ B , is uniquely determined. Two linear manifolds, say .M1 and .M2 , in .B are said to be complementary in .B if .M1 ∩M2 = {0} and .M1 +M2 := {f ∈ B : f = f1 +f2 , f1 ∈ M1 , f2 ∈ M2 } = B and we say that .B is a direct sum of .M1 and .M2 and we denote it by .B = M1 ⊕ M2 . If moreover .M1 and .M2 are closed subspaces of .B, then there exists a bounded projector .PM1 such that .PM1 B = M1 and .(I − PM1 )B = M2 , where I is the identity operator. An operator A is said to be closed if the relations .(fn ) ⊂ D(A), .fn − f  → 0, and .gn = Afn , with .gn − g → 0, imply that .f ∈ D(A) and .Af = g. A densely defined closed operator A is said to be invertible reducible, if B = N(A) ⊕ R(A).

.

A

Markov Chains

187

Let us consider a Markov chain, say .Xn , .n ≥ 0, with state space .(E, E ) and transition probability .P (x, A), .x ∈ E, .A∈ E . We will denote by the same letter P the transition operator, that is, .Pf (x) = E P (x, dy)f (y), with .f ∈ B, the Banach space of real bounded measurable functions defined on E, and .P 0 = I , the identity operator in .B. For the space .M(E) of finite signed measures with norm .·, let us introduce the dual space R of functions on .f E , the space of measurable functions with respect to .E , with the finite norm .

f  := sup{|μf | : μ ≤ 1}.

(A.24)

Introduce the space of transition kernels Q such that .MQ ⊂ M(E), with .

Q := sup{μQ : μ ≤ 1} < ∞.

(A.25)

Definition A.12 A Markov chain X is said to be uniformly ergodic with respect to the norm .·, if there exists a transition kernel .Π , such that P n −→ Π,

.

n → ∞,

in the induced operator norm (A.25), that is, .

  lim sup (P n − Π )f  = 0,

n→∞ f ≤1

f ∈ B.

For a uniformly ergodic Markov chain, the operator .Q = P − I , where I is the identity operator, is reducible-invertible (see, e.g., [90–92]) B = N(Q) ⊗ R(Q),

.

dimN(Q) = 1.

The null space consists of functions .f (x) = constant. The stationary projector, defined by 

ρ(dy)f (y)1(x) = f1(x),

Πf (x) =

.

f :=

E

 ρ(dy)f (y), E

is the projector onto the null space .N(Q). The subspace of values .R(Q) is closed. Definition A.13 A stochastic transition kernel .Π on .(E, E ) is said to be a stationary projector of a transition kernel P on .(E, E ), if Π2 = Π = P Π = ΠP

.

and .μ = μΠ provided that .μ = μP , .μ ∈ M(E).

188

A Markov Chains

Theorem A.1 Let a Markov chain X be uniformly ergodic with respect to the norm ·. Then the limit kernel .Π , in Definition A.12, is the stationary projector of the kernel P .

.

The potential operator .R0 of a uniform ergodic Markov chain with transition operator P and stationary projector .Π is defined by R0 = (P − I + Π )−1 − Π =



.

(P n − Π ).

n≥1

The operator .R0 is bounded. This follows from the uniform convergence of the series   (P n − Π )f  < ∞, f ∈ B, f  ≤ 1. . n≥1

For further reading on Markov chains the following references are of interest [21, 24, 47, 53, 62, 83, 84, 116, 182].

Bibliography

1. Aase, K. (1988). Contingent claims valuation when the securities price is a combination of an Ito process and a random point process. Stochastic Processes and their Applications, 28, 185–220. 2. Adams, R. (1979) Sobolev spaces. New York, NY: Academic Press. 3. Altman, E., & Shwartz, A. (1991). Markov decision problems and state-action frequencies. SIAM Journal of Control and Optimization, 29(4), 786–809. 4. Anisimov, V. V. (1977). Switching processes. Cybernetics, 13(4), 590–595. 5. Anisimov, V. V. (2008). Switching processes in queueing theory. iSTE. UK: J. Wiley. 6. Anisimov, V. V. (1991). Averaging principle for switching recurrent sequences. Theory of Probability and Mathematical Statistics, 45, 1–8. 7. Anisimov, V. V. (1992). Averaging principle for switching processes. Theory of Probability and Mathematical Statistics, 46, 1–10. 8. Anselone, P. M. (1960). Ergodic theory for discrete semi-Markov chains. Duke Mathematical Journal, 27(1), 33–40. 9. Arjas, E., & Korolyuk, V. S. (1980). Stationary phase merging of Markov renewal processes (in Russian). DAN of Ukraine, 8, 3–5. 10. Ascione G., & Cuomo, S. (2021). A sojourn-based approach to semi-Markov reinforcement learning. arXiv:2021.06827v1 [math.PR]. 11. Asmussen, S. (1987). Applied probability and queues. Chichester: Wiley. 12. Barbu, V., & Limnios, N. (2008). Semi-Markov Chains and Hidden Semi-Markov Models. Toward Applications – Their use in Reliability and DNA Analysis (Lecture Notes in Statistics, Vol. 191). New York: Springer. 13. Bertsekas, D., & Shreve, S. (1996). Stochastic optimal control: The discrete-time case. Belmont, MA: Athena Science Publication. 14. Beutler, F., & Ross, K. (1985). Optimal policies for controlled Markov chains with a constraint. Journal of Mathematical Analysis and Applications, 112, 236–252. 15. Bhattacharya, R. N., & Waymire, E. C. (1990). Stochastic processes with applications. New York, NY: Wiley. 16. Billingsley, P. (1999). Convergence of probability measures. New York, NY: Wiley. 17. Björk, T. (1998). Arbitrage in continuous time. New York, NY: Oxford University Press Inc. 18. Bobrowski, A. (2005). Functional analysis for probability and stochastic processes. Cambridge: Cambridge University Press. 19. Borkar, V. S. (2003). Dynamic programming for ergodic control with partial observations. Stochastic Processes and Their Applications, 103, 293–310.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0

189

190

Bibliography

20. Borkar, V. S. (2008). Stochastic approximation – A dynamical systems viewpoint. India: Oxford University Press/Hindustan Book Agency. 21. Borovkov, A. A. (1998). Ergodicity and stability of stochastic processes. Chichester: Wiley. 22. Bouzebda, S., & Limnios, N. (2013). On general bootstrap of empirical estimator of a semiMarkov kernel with applications. Journal of Multivariate Analysis, 116, 52–62. 23. Boussement, M., & Limnios, N. (2004). Markov decision processes with asymptotic average failure rate constraint. Communications in Statistics – Theory and Methods, 33(7), 1689– 1714. 24. Breiman, L. (1968). Probability. Mass: Addison-Wesley. 25. Cartea, A., Jaimungal, S., & Penalva, J. (2015). Algorithmic and high-frequency trading. Cambridge, UK: Cambridge University Press. 26. Chabanyuk, Y., Nikitin, A., & Khimka, U. (2020). Asymptotic analysis for complex evolutionary systems with Markov and Semi-Markov switching using approximation schemes. iSTE. London: J. Wiley. 27. Chiquet, J., Limnios, N., & Eid, M. (2009). Piecewise deterministic Markov processes applied to fatigue crack growth modelling. Journal of Statistical Planning and Inference, 139, 1657– 1667. 28. Chow, Y., Robbins, H., & Siegmund, D. (1971). The theory of optimal stopping rules. Boston: Houghton Miffin Company. 29. Chung, K. L. (1982). Lectures from Markov processes to Brownian motion. New York, NY: Springer. 30. Cohen, J. E. (1979). Random evolutions in discrete and continuous time. Stochastic Processes and Their Applications, 9, 245–251. 31. Comets, F., & Pardoux, E. (Eds.) (2001). Milieux Aléatoires. Société Mathématique de France, No 12. 32. Cox, J. C., Ross, R. A., & Rubinstein, M. (1976). Option pricing: A simplified approach. Journal of Financial Economics, 7, 229–263. 33. Daley, D. J., & Gani, J. (1999). Epidemic modelling. Cambridge: Cambridge University Press. 34. Dautray, R., Cessenat, M., Ledanois, G., Lions, P.-L., Pardoux, E., & Sentis, R. (1989). Méthodes Probabilistes pour les Equations de la Physique. Paris: Eyrolles. 35. Da Prato, G. (2006). An introduction to infinite dimensional analysis. Berlin: Springer. 36. Davis, M. H. A. (1993). Markov models and optimization. New York: Chapman & Hall. 37. Dellacherie, C., & Meyer, P. A., (1982). Probabilities and potential. Amsterdam: B. NorthHolland. 38. Doob, J. L. (1954). Stochastic processes. New York: Wiley. 39. Dudley, R. M. (1989). Real analysis and probability. New York: Wadsworth. 40. Dunford, N., & Schwartz, J. (1958). Linear operators. General theory. Interscience, New Jercy: Wiley. 41. Dynkin, E. (1965). Markov processes. New York: Springer. 42. Elliott, R. J., Limnios, N., & Swishchuk, A. (2013). Filtering hidden Semi-Markov chains. Statistics and Probability Letters, 83(9), 2007–2014. 43. Elliott, R., Swishchuk, A., & Zhang, I. Y. (2017). A semi-martingale representation for semiMarkov chain with application to finance. Theory of Probability and Mathematical Statistics, 96, 49–60. 44. Esquível, M. L., Krasii, N. P., & Guerreiro, G. R. (2021). Open Markov type population models: From discrete to continuous time. Mathematics, 9, 1496. 45. Ethier, S. N., & Kurtz, T. G. (1986). Markov processes: Characterization and convergence. New York: J. Wiley. 46. Ethier, S. N., Feng, J., & Stockbridge, R. H. (Eds.) (2008). Markov processes and related topics – A Festrischrift for Tomas G. Kurtz. Beachwood, OH: IMS. 47. Feller, W. (1966). An introduction to probability theory and its applications (Vols. 1 and 2). New York, NY: J. Wiley. 48. Fleming, W. H., & Rishel, R. W. (1975). Deterministic and stochastic optimal control. New York, NY: Springer.

Bibliography

191

49. Fleming, W. H., & Mete Soner, H. (1993). Controlled Markov processes and Viscosity solutions. New York, NY: Springer. 50. Freidlin, M. I. (Ed.) (1994). The Dynkin Festschrift: Markov processes and their applications. Boston: Birkäuser, 51. Freidlin, M. I. (1996). Markov processes and differential equations: asymptotic problems (Lectures in Mathematics, ETH Zürich). Basel: Birkhäuser 52. Freidlin, M. I., & Wentzell, A. D. (1998). Random perturbations of dynamical systems (2nd ed.). New York: Springer. 53. Fristedt, B., & Gray, L. (1997). A modern approach to probability theory. Boston: Birkhäuser. 54. Gihman, I. I., & Skorohod, A. V. (1974). Theory stochastic processes (Vols. 1, 2, and 3). Berlin: Springer. 55. Glynn, P. W. (1990). Diffusion approximation. In D. P. Heyman, & M. J. Sobel (Eds.), Handbook in operations research and management science (Stochastic Models, Vol. 2, pp. 145–198). Amsterdam: Elsevier Science Publishers. 56. Grabski, F. (2015). Semi-Markov processes: Applications in reliability and maintenance. Amsterdam: Elsevier. 57. Gregoratti, M. (2010). Dilations à la quantum probability of Markov evolutions in discrete time. Theory of Probability and Its Applications, 54(1), 140–150. 58. Griego, R., & Hersh, R. (1969). Random evolutions, Markov chains, and systems of partial differential equations. Proceedings of the National Academy of Sciences of the United States of America, 62, 305–308. 59. Griego, R., & Hersh, R. (1971). Theory of random evolutions with applications to partial differential equations. Transactions of the American Mathematical Society, 156, 405–418. 60. Gut, A. (1988). Stopped random walks. New York: Springer. 61. Hall, P., & Heyde C. (1980). Martingale limit theorems and its applications. New York: Academic Press. 62. Hernández-Lerma, O., & Lasserre, J. B. (2003). Markov chains and invariant probabilities. Basel: Birkhäuser. 63. Hernández-Lerma, O., & Lasserre, J. B. (1996). Discrete-time Markov control processes. New York, NY: Springer. 64. Hersh, R. (1974). Random evolutions: A survey of results and problems. Rocky Mountain Journal of Mathematics, 4, 443–477. 65. Hersh, R. (2003). The birth of random evolutions, Mathematical Intelligencer, 25(1), 53–60. 66. Hersh, R., & Papanicolaou, G. (1972). Non-commuting random evolutions, and an operatorvalued Feynman-Kac formula. Communications on Pure and Applied Mathematics, 30, 337– 367. 67. Hersh, R., & Pinsky, M. (1972). Random evolutions are asymptotically Gaussian. Communications on Pure and Applied Mathematics, 25, 33–44 68. Heyde, C. C., Prohorov, Yu. V., Pyke, R., & Rachev, S. T. (Eds.) (1996). Athens conference on applied probability and time series, LNS 114. New York: Springer. 69. Hille, E., & Phillips, R. (1962). Functional analysis and semigroups. Providence, RI: AMS. 70. Howard, R. (1971). Dynamic probabilistic systems (Vol. 2). New York: J. Wiley. 71. Ito, K., & McKean, M. P., Jr. (1996). Diffusion processes and their sample paths. Berlin: Springer. 72. Ja´skiewicz, A., & Nowak, A. (2007). Average optimality for semi-Markov control processes. Morfismos, 11(1), 15–36. 73. Jacod, J., & Shiryaev, A. N. (1987). Limit theorems for stochastic processes. Berlin: Springer. 74. Jefferies, B. (1996). Evolution processes and the Feynman-Kac formula. Dordrecht: Kluwer. 75. Janssen, J. (Ed.) (1986). Semi-Markov models – Theory and applications. New York: Springer. 76. Janssen, J., & Limnios, N. (1999). Semi-Markov models and applications. Dordrecht: Kluwer Academic Publisher. 77. Janssen, J., & Manca, R. (2007). Semi-Markov risk models for finance, insurance and reliability. New York: Springer.

192

Bibliography

78. Kabanov, Y., & Pergamenshchiskov, S. (2003). Two-scale stochastic systems: Asymptotic analysis and control. Berlin: Springer. 79. Karatzas, I., & Shreve, S. (1998). Methods of mathematical finance. Berlin: Springer. 80. Karlin, S., & Taylor, H. M. (1981). A second course in stochastic processes. San Diego: Academic Press. 81. Kartashov, N. V. (1996). Strong stable Markov chains. VSP Utrecht, TBiMC Kiev. Zutphen: VSP. 82. Keepler, M. (1998). Random evolutions processes induced by discrete time Markov chains. Portugaliae Mathematica, 55(4), 391–400 83. Keelson, J. (1979). Markov chains models – Rarity and exponentiality. New York: Springer. 84. D. Khoshnevisan, (2002). Multiparameter processes. New York: Springer. 85. Kifer, Y. (1988). Random perturbations of dynamical systems. Boston: Birkhäuser. 86. Kimura, M. (1964). Diffusion models in population genetics. Journal of Applied Probability, 1, 177–232. 87. Klebaner, F. C. (1998). Introduction to stochastic calculus with applications. London: Imperial College Press. 88. Koroliouk, D. (2020). Dynamics of statistical experiments, iSTE. London: J. Wiley. 89. Koroliouk, D., Samoilenko, I. (2021). Random evolutionary systems: Asymptotic properties and large deviations, ISTE. London: J. Wiley. 90. Koroliuk, V. S., & Limnios N. (2005). Stochastic systems in merging phase space. Singapore: World Scientific. 91. Korolyuk, V. S., & Swishchuk, A. (1995). Evolution of system in random media. Boca Raton: CRC Press. 92. Korolyuk, V. S., & Turbin, A. (1993). Mathematical foundations of the states lumping of large systems. Dordrecht: Kluwer. 93. Krylov, N., & Bogolyubov, N. (1947). Introduction to non-linear mechanics. Princeton: Princeton University Press. 94. Kuo, H.-H. (2006). Introduction to stochastic integration. USA: Springer. 95. Kushner, H. J. (1971). Introduction to stochastic control theory. New York: Holt, Rinehart, Winston. 96. Kushner, H. J. (1990). Weak convergence methods and singular perturbed stochastic control and filtering problems. Boston: Birkhäuser. 97. Lamperti, J. (1977). Stochastic processes. New York: Springer. 98. Lapeyre, B., Pardoux, E., & Sentis, R. (1998). Méthodes de Monte-Carlo pour les equations de transport et de diffusion. Paris: Springer. 99. Levin, K., & Betancourt, B. (2022). Fast generation of exchangeable sequence of clusters data. arXiv: 2209.02844v1 [math.ST]. 100. Limnios, N. (2004). Functional central limit theorems for empirical estimators of semiMarkov processes. Journal of Nonparametric Statistics, 16(1–2), 13–18. 101. Limnios, N. (2011). Discrete-time random evolutions – Difference equations and additive functionals. Communications in Statistics – Theory and Methods, 40(19–20), 1–11. 102. Limnios, N., & Wu B. (2021). Estimation of stationary probability of semi-Markov chains. Statistical Inference for Stochastic Processes. https://doi.org/10.1007/s11203-021-09255-3 103. Limnios, N., & Opri¸san, G. (2001). Semi-Markov processes and reliability. Boston: Birkhäuser. 104. Limnios, N., & Swishchuk, A. (2013). Discrete-time semi-Markov random evolutions and their applications. Advances in Applied Probability, 45(1), 214–240. 105. Limnios, N., & Swishchuk, A. (2020). Discrete-time Semi-Markov random evolutions in asymptotic reduced random media with applications. Mathematics, 8, 923. https://doi.org/ 10.3390/math8060963 106. Limnios, N., & Yarovaya, E. (2019). Diffusion approximation of near critical branching processes in fixed and random environment. Stochastic Models, 35(2), 209–220. https://doi. org/10.1080/15326349.2019.1578240

Bibliography

193

107. Limnios, N., & Yarovaya, E. (2020). Diffusion approximation of Branching processes in semiMarkov environment. Methodology and Computing in Applied Probability, 22, 1583–1590. https://doi.org/10.1007/s11009-020-09814-w 108. Lindvall, T. (1992). Lectures on the coupling method. New York: J. Wiley. 109. Liptser, R. Sh., & Shiryayev, A. N. (1989). Theory of martingales. Dordrecht: Kluwer Academic Publishers. 110. Logachov, A., Mogulskii, A., Prokopenko, E., & Yambartsev, A. (2021). Local theorems for (multidimensional) additive functionals of semi-Markov chains. Stochastic Processes and Their Applications, 137, 149–166. 111. Maxwell, M., & Woodroofe, M. (2000). Central limit theorems for additive functionals of Markov chains. Annals of Probability, 28(2), 713–724. 112. Merton, R. (1969). Lifetime portfolio selection under uncertainty: The continuous-time case. The Review of Economics and Statistics, 51(3), 247–257. 113. Merton, R. (1971). Optimum consumption and portfolio rules in a continuous-time model. Journal of Economic Theory, 3, 373–413. 114. Merton, R. (1990). Continuous-time finance. Cambridge: Basil Blackwell Inc. 115. Métivier, M. (1982). Semimartingales. A course on stochastic processes. Berlin: Walter de Gruyter. 116. Meyn, S. P., & Tweedie, R. L. (1993). Markov chains and stochastic stability. New York: Springer. 117. Mode, C. J., & Sleeman, C. K. (2000). Stochastic processes in epidemiology. HIV/AIDS, other infectious diseases and computers. Singapore: World Scientific. 118. Murdock, J. A. (1999). Perturbations: theory and methods, Philadelphia: SIAM, Classics in Applied Mathematics. 119. Neuts, M. N. (1981). Matrix-geometric solutions in stochastic models: An algorithmic approach. Baltimore: Johns Hopkins University Press. 120. Nummelin, E. (1984). General irreducible Markov chains and non-negative operators. Cambridge: Cambridge University Press. 121. Orey, S. (1971). Lecture notes in limit theorems for Markov chains transition probabilities. London: Van Nostrand. 122. Pachon, A., Polito, F., & Ricciuti, C. (2021). On discrete-time semi-Markov processes. Discrete and Continuous Dynamical Systems Series B, 26(3), 1499–1529. 123. Papanicolaou, G. (1987). Random media. Berlin: Springer. 124. Papanicolaou, G., Stroock, D. W., & Varadhan, S. R. S. (1977). Martingale approach to some limit theorems. In D. Ruelle (Ed.), Statistical mechanics, dynamic systems and the duke turbulence conference (Vol. 3). Duke University Series. 125. Papanicolaou, G., Kohler, W., & White, B. (1991). Random media (Lectures in Applied Mathematics, Vol. 27). Philadelphia: SIAM. 126. Parzen, E. (1999). Stochastic processes. Philadelphia: SIAM Classics. 127. Petersson, M. (2016). Perturbed discrete time stochastic models. PhD Thesis, Department of Mathematics, Stockholm University. 128. Petrov, V. V. (1995). Limit theorems of probability theory. Oxford: Clarendon. 129. Pinsky, M. (1991). Lectures on random evolutions. Singapore: World Scientific. 130. Pliska, S. (1998). Introduction to mathematical finance: Discrete time models. London: Blackwell Publications Ltd. 131. Pollard, D. (1984). Convergence of stochastic processes. New York: Springer. 132. Pyke, R. (1961). Markov renewal processes: Definitions and preliminary properties. Annals of Mathematical Statistics, 32, 1231–1242 133. Pyke, R. (1961). Markov renewal processes with finitely many states. Annals of Mathematical Statistics, 32, 1243–1259. 134. Pyke, R., & Schaufele, R. (1964). Limit theorems for Markov renewal processes. Annals of Mathematical Statistics, 35, 1746–1764. 135. Rachev, S., & Rucshendorf, L. (1994). Model for option prices. Theory of Probability and Its Applications, 39(1), 120–152.

194

Bibliography

136. Revuz, D. (1975). Markov chains. Amsterdam: North-Holland Publishing Company. 137. Rogers, L.C.G., & Williams, D. (1994, 1996). Diffusions, Markov processes, and martingales (Vol. 1: Foundations, Vol. 2: Itô calculus). Chichester: J. Wiley. 138. Ross, R., & Hudson, H.P. (1917). An application of the theory of probabilities to the study of a priori pathometry, II, III. Proceedings of the Royal Society of London. Series A, 93, 212–225; 225–240. 139. Rudin, W. (1991). Functional analysis. New York: McGraw-Hill. 140. Shiryaev, A. N. (1999). Essentials of stochastic finance: facts, models, theory. Singapore: World Scientific. 141. Shiryaev, A., Kabanov, Yu., Kramkov, D., & Melnikov, A. (1994). Towards the theory of pricing of options of both European and American types. I. Discrete time. Theory of Probability and Its Applications, 39(1), 23–79 142. Shreve, S. (2005). Stochastic calculus for finance 1: The binomial asset pricing model. New York: Springer. 143. Shurenkov, V. M. (1984). On the theory of Markov renewal. Theory of Probability and Its Applications, 19(2), 247–265. 144. Skorokhod A.V. (1984). Random linear operators. Dordrecht: R. Riedel-Kluwer. 145. Skorokhod, A. V. (1988). Stochastic equations for complex systems. Dordrecht: R. RiedelKluwer. 146. Skorokhod, A. V. (1989). Asymptotic methods in the theory of stochastic differential equations (Vol. 78). Providence: AMS. 147. Skorokhod, A. V. (1991). Random processes with independent increments. Dordrecht: Kluwer. 148. Skorokhod, A. V., Hoppensteadt, F. C., & Salehi, H. (2002). Random perturbation methods with applications in science and engineering. New York: Springer. 149. Silvestrov, D. S. (2004). Limit theorems for randomly stopped stochastic processes. Series: Probability and its Applications. New York: Springer. 150. Silvestrov, D. S., & Silvestrov S. (2017). Nonlinearly perturbed semi-Markov processes. New York: Springer. 151. Sobolev, S. (1991). Some applications of functional analysis in mathematical physics (Vol. 90, 3rd ed.). USA: American Mathematical Society. 152. Soubeyrand, S. (2016). Construction of semi-Markov genetic-space-time SFEIR models and inference. JSFS, 157(1), 129–152 153. Stroock, D. W., & Varadhan, S. R. S. (1979). Multidimensional diffusion processes. Berlin: Springer. 154. Sviridenko, M. N. (1986). Martingale approach to limit theorems for semi-Markov processes. Theory of Probability and Its Applications, 34(3) 540–545. 155. Svishchuk, M. (2011). Endemic SIR Model in Random Media. Working paper, 8p. 156. Swishchuk, A. V. (2000). Random evolutions and their applications. New trends. Dordrecht: Kluwer. 157. Swishchuk, A. V. (1997). Random evolutions and their applications (Vol. 408). Dordrecht: Kluwer. 158. Swishchuk, A. (2019). Inhomogeneous random evolutions and their application. Boca Raton: CRC Press. 159. Swishchuk, A. V., & Islam M. S. (2010). The geometric Markov renewal processes with application to finance. Stochastic Analysis and Applications, 29, 4. 160. Swishchuk, A. V., & Islam, M. S. (2010). Diffusion approximations of the geometric Markov renewal processes and option price formulas. International Journal of Stochastic Analysis, 2010, 21. Article ID 347105. https://doi.org/10.1155/2010/347105 161. Swishchuk, A. V., & Islam, M. S. (2013). Normal deviation and Poisson approximation of GMRP. Communications in Statistics – Theory and Methods, 42(8), 1488–1501. 162. Swishchuk, A. V., & Islam, M. S. (2013). Random dynamical systems in finance. Boca Raton, FL: CRC Press.

Bibliography

195

163. Swishchuk, A. V., & Limnios, N. (2011). Optimal stopping of GMRP and pricing of European and American options. In Proceedings of the 15th International Congress on Insurance: Mathematics and Economics (IME 2011), 14–17 June 2011, Trieste, Italy. 164. Swishchuk, A., & Limnios, N. (2021). Controlled discrete-time semi-Markov random evolutions and their applications. Mathematics, 9, 158. https://doi.org/10.3390/math9020158 165. Svishchuk, M., & Swishchuk, A. (2012). Merging of Endemic SIR Model in semi-Markov Random Media. Working paper, 12p. 166. Swishchuk, A., & Wu, J. (2003). Evolution of biological systems in random media: limit theorems and stability. Dordrecht: Kluwer. 167. Swishchuk, A., & Wu, J. (2003). Limit theorems for difference equations in random media with applications to biological systems. Random Operators and Stochastic Equations, 11(1), 25–76. 168. Swishchuk, A., Svishchuk, M., & Limnios, N. (2016). Stochastic stability of vector SDE with applications to a stochastic epidemic model. International Journal of Pure and Applied Mathematics, 106(3), 801–839. 169. Swishchuk, A. V., Zhuravitsky, D. G., & Kalemanova, A. V. (2000). Analogue of BlackScholes formula for .(B, S, X)-securities market with jumps. Ukrainian Mathematical Journal, 52(3), 489–497. 170. Van der Hoek, J., & Elliott, R. J. (2018). Introduction to hidden semi-Markov models (Vol. 445). London Mathematical Society, LNS. London: Cambridge University Press. 171. Vassiliou, P.-C., & Georgiou, A. C. (Eds.) (2021). Markov and Semi-Markov Chains, Processes, Systems and Emerging Related Fields. Special volume, Mathematics, MDPI. 172. Vega-Amaya, O., & Luque-Vásquez, F. (2000). Sample-path average cost optimality for semi-Markov control processes on Borel spaces: unbounded cost and mean holding times. Applicationes Mathematicae, 27(3), 343–367. 173. Votsi, I., Limnios, N., Papadimitriou, E., & Tsaklidis, G. (2019). Earthquake statistical study by a multistate modeling approach. Iste. London: J. Wiley. 174. Watanabe, H. (1984). Diffusion approximation of some stochastic difference equations. Advances in Applied Probability, 7, 439–546. 175. Watkins, J. (1984). A central limit theorem in random evolution. Annals of Probability, 12, 480–514. 176. Webb, G., Blaser, M. J., Zhu, H., Ardal, S., & Wu, J. (2003). Critical role of nosocomial transmission in the Toronto SARS outbreak, Preprint. 177. Whitt, W. (2002). Stochastic-process limits: An introduction to stochastic-process limits and their applications to queues. New York: Springer. 178. Yin, G. G., & Zhang, Q. (2005). Discrete-time Markov chains. Two-time-scale methods and applications. New York: Springer. 179. Yosida, K. (1980). Functional analysis. Berlin: Springer. 180. Yu, P. S. (1977). On accuracy improvement and applicability conditions of diffusion approximation with applications to modelling of computer systems. Technical Report No. 129, Digital Systems Lab., Stanford University. 181. Zariphopoulou, Th. (2002). Stochastic control methods in asset pricing, Chap. 12. In D. Kannan & V. Lakshmikantham (Eds.), Handbook of stochastic analysis and applications (Statistics: Textbooks and monographs, Vol. 163). New York: Marcel Dekker. 182. Zhenting, H., & Qingfeng, G. (1988). Homogeneous denumerable Markov processes. Berlin: Springer and Beijing: Science Press.

Index

A Additive functional, 50, 92 controlled -, 111 Alternating renewal chain, 25 American option, 168 Average approximation, 86 Averaging, 56

B Banach space separable -, 47 weak convergence -, 12 Bochner integral, 2 Brownian motion, 10

C Compact relatively -, 62 weakly -, 67 Compact containment condition, 60 Compact containment criterion, 13 Compactly embedded space, 57 Compensating operator, 31

D Diffusion approximation, 57, 86 Discount capital, 166 Discount price, 166 Discrete-time semi-Markov random evolutions (DTSMRE) controlled -, 101, 103, 107

DPE/HJB equation, 116 Dynamical system, 52, 94 controlled -, 114 Dynamic programing controlled models, 105 equation, 115

E Ergodic Markov chain uniformly -, 186 Ergodic merging, 84 European option, 172

G Geometric Markov renewal chains (GMRC), 159 martingale properties - , 162 optimal stopping -, 157 Geometric Markov renewal process (GMRP) controlled -, 112

I Inter-jump time, 21 Invariant measure, 184

J Jump times, 20

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 N. Limnios, A. Swishchuk, Discrete-Time Semi-Markov Random Evolutions and Their Applications, Probability and Its Applications, https://doi.org/10.1007/978-3-031-33429-0

197

198 M Markov chain, 24, 177 embedded -, 21 Markov process, 38 Markov renewal chain, 21, 31 geometric -, 52, 95 equation, 27 function, 27 process, 21 Theorem, 27 Martingale, 3 characterization, 32 problem, 11 quadratic variation of a -, 3 square integrable -, 3 weak -, 3 Merging state space, 36 Merton problem, 118 solution -, 118, 123 N Normal deviations, 58, 89 O Operator perturbed -, 15 potential -, 14 projection -, 13 reducible invertible -, 13 Optimal stopping pricing, 157 rule, 160 time, 160 P Performance criterion, 120 Perturbation problem, 39 Polish space, 12 preface, v Projection operator, 47 R Random evolution average -, 45 discrete-time -, 43

Index Rate of convergence, 59, 124 Reduced random media, 83 Relative compactness, 60 Riskless asset, 160

S Semigroup, 5 contraction-, 7 generator-, 7, 8 strongly continuous-, 7 Semi-Markov chain, 19, 22 kernel, 21 cumulative -, 21 Semi-Markov process, 6 Singular perturbing problem, 60 Skorohod space, 55 Stationary phase, 34 Stochastic system in series scheme, 53 Sub-stochastic kernel, 179 Survival function, 30

T Tightness, 60 Time-scaling, 57

U Ulam theorem, 12 U-statistics, 97 Utility function, 119

V Value function, 120

W Weak convergence, 56 Weakly compact processes, 12 Wiener process, 10