Reconstruction of chaotic signals with applications to chaos-based communications 9812771131, 9789812771131

Citation preview

Reconstruction nstruction of

Chaotic Signals with Applications to Chaos-Based Communications

Jiu Chao Feng South China University ofTechnology, Guangzhou, China

Chi Kong Tse The Hong Kong Polytechnic University, Hong Kong

World Scientific NEW JERSEY· LONDON· SINGAPORE· BEIJING· SHANGHAI· HONG KONG· TAIPEI· CHENNAI

Published by

jl'ii/'!7c"j: W!\&ft (Tsinghua University Press) -I tfJ\ jj!j i/'! ａＢｴｾＳｔＷ｣｛ｬ＠ A ｾ＠ ｾｬ［ａＱｕＺ＠

100084

and World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link. Singapore 596224 USA office: 27 Warren Street, Suite 401-402. Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

ill i'i': {1§ ｾ＠ fl'J 4l: fiJ Ez ｾＺｴｅ＠ cljk Till i'i': 81 jffi {151' 81 @. ffi ｾ＠ Reconstruction of Chaotic Signals with Applications (0 Chaos-Based Communicacions/lllJ;7\11'lL iMi'I'lxjIJet,-)t.M: lfii/'!:;J(''¥:WhiRH, 2007.11 ISBN 978-7-302-12120-6

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

RECONSTRUCTION OF CHAOTIC SIGNALS WITH APPLICATIONS TO CHAOS-BASED COMMUNICATIONS Copyright © 2008 by Tsinghua University Press and World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof: may IlOt be reproduced ill anv j(mn or by any meallS, electronic or mechanical, including photocopying, recording or any information storage and retrieval system /lOW kllown or to be invented, without written pemlissionjrom the Publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.

ISBN-13 978-981-277-113-1 ISBN-l0 981-277-113-1

Printed in Singapore by World Scientific Printers

Preface

The study of time series has traditionally been within the realm of statistics. A large number of both theoretical and practical algorithms have been developed for characterizing, modeling, predicting and filtering raw data. Such techniques are widely and successfully used in a broad range of applications, e.g., signal processing and communications. However, statistical approaches use mainly linear models, and are therefore unable to take advantage of recent developments in nonlinear dynamics. In particular, it is now widely accepted that even simple nonlinear deterministic mechanisms can give rise to complex behavior (i.e., chaos) and hence to complex time series. Conventional statistical time-series approaches are unable to model or predict complex time series with a reasonable degree of accuracy. This is because they make no use of the fact that the time series has been generated by a completely deterministic process, and hence ascribe most of the complexity to random noise. Furthermore, such approaches cannot yield much useful information about the properties of the original dynamical system. Fortunately, a remarkable result due to Takens shows that one can reconstruct the dynamics of an unknown deterministic finite-dimensional system from a scalar time series generated by that system. Takens' theorem is actually an extension of the classical Whitney's theorem. It is thus concerned with purely deterministic autonomous dynamical systems and the framework that it provides for time series analysis is unable to incorporate any notion of random behavior. This means that the process of reconstruction is outside the scope of statistical analysis because any such analysis would require a stochastic model of one kind or another as its starting point. This is reflected in common practice, where reconstruction is seen as a straightforward algorithmic procedure that aims to recover properties of an existing, but hidden, system.

Reconstruction of Chaotic Signals with Applications to Chaos-Based Communications

The problem of reconstructing signals from noisy corrupted time series arises in many applications. For example, when measurements are taken from a physical process suspected of being chaotic, the measuring device introduces error in the recorded signal. Alternatively, the actual underlying physical phenomenon may be immersed in a noisy environment, as might be the case if one seeks to detect a low-power (chaotic) signal (possibly used for communications) that has been transmitted over a noisy and distorted channel. In both cases, we have to attempt to purify or detect a (chaotic) signal from noisy and/or distorted samples. Separating a deterministic signal from noise or reducing noise in a noisy corrupted signal is a central problem in signal processing and communications. Conventional methods such as filtering make use of the differences between the spectra of the signal and noise to separate them, or to reduce noise. Most often the noise and the signal do not occupy distinct frequency bands, but the noise energy is distributed over a large frequency interval, while the signal energy is concentrated in a small frequency band. Therefore, applying a filter whose output retains only the signal frequency band reduces the noise considerably. When the signal and noise share the same frequency band, the conventional spectrumbased methods are no longer applicable. Indeed, chaotic signals in the time domain are neither periodic nor quasi-periodic, and appear in the frequency domain as wide "noise-like" power spectra. Conventional techniques used to process classic deterministic signals will not be applicable in this case. Since the property of self-synchronization of chaotic systems was discovered in 1990 by Pecora and Carroll, chaos-based communications have received a great deal of attention. However, despite some inherent and claimed advantages, communicating with chaos remains, in most cases, a difficult problem. The main obstacle that prevents the use of chaotic modulation techniques in real applications is the very high sensitivity of the demodulation process. Coherent receivers which are based on synchronization suffer from high sensitivity to parameter mismatches between the transmitter and the receiver and even more from signal distortion/contamination caused by the communication channel. Non-coherent receivers which use chaotic signals for their good decorrelation properties have been shown to be more robust to channel noise; however, as iv

Preface

they rely mainly on (long-term) decorrelation properties of chaotic carriers, their performances are very close to those of standard non-coherent receivers using pseudo-random or random signals. In both cases, coherent and non-coherent, it has been shown that noise reduction (or signal separation) can considerably improve the performance of chaos-based communication systems. The aforementioned problems can be conveniently tackled if signals can be reconstructed at the receiving end. Motivated by the general requirements for chaotic signal processing and chaos-based communications, this book addresses the fundamental problem of reconstruction of chaotic signals under practical communication conditions. Recently, it has been widely recognized that artificial neural networks endow some unique attributes such as universal approximation (input-output mapping), the ability to learn from and adapt to their environments, and the ability to invoke weak assumptions about the underlying physical phenomena responsible for the generation of the input data. In this book, the technical approach to the reconstruction problem is based on the use of neural networks, and the focus is the engineering applications of signal reconstruction algorithms. We begin in Chapter 1 by introducing some background information about the research in signal reconstruction, chaotic systems and the application of chaotic signals in communications. The main purpose is to connect chaos and communications, and show the potential benefits of using chaos for communications. In Chapter 2 we will review the state of the art in signal reconstruction, with special emphasis laid on deterministic chaotic signals. The Takens' embedding theory will be reviewed in detail, covering the salient concepts of reconstructing the dynamics of a deterministic system from a higher-dimensional reconstruction space. The concepts of embedding dimension, embedding lag (time lag), reconstruction function, etc., will be discussed. Two basic embedding methods, namely topological and differential embeddings, will be described. Some unsolved practical issues will be discussed. A thorough review of the use of neural networks will be given in Chapter 3. Two main types of neural networks will be described in detail, namely, the radial-basis-function neural networks and the recurrent neural networks. The background theory,

v

Reconstruction of Chaotic Signals with Applications to Chaos-Based Communications

network configurations, learning algorithms, etc., will be covered. These networks will be shown suitable for realizing the reconstruction tasks in Chapters 7 and 8. In Chapter 4 we will describe the reconstruction problem when signals are transmitted under ideal channel conditions. The purpose is to extend the Takens' embedding theory to time-varying continuous-time and discrete-time systems, i.e., nonautonomous systems. This prepares the readers for the more advanced discussions given in the next two chapters. In Chapter 5, we will review the Kalman filter (KF) and the extended Kalman filter (EKF) algorithms. In particular, we will study a new filter, i.e., the unscented Kalman filter (UKF). The issue for filtering a chaotic system from noisy observation by using the UKF algorithm will be investigated. A lot of new finding of the UKF algorithm will be demonstrated in this chapter. In Chapter 6, we will apply the UKF algorithm to realize the reconstruction of chaotic signals from noisy and distorted observation signals, respectively. Combining the modeling technique for signal with the UKF algorithm, the original UKF algorithm will be expanded to filter timevarying chaotic signals. We will also address the issues of blind equalization for chaos-based communication systems. Our start point is to make use of the UKF algorithm and the modeling technique for signal, and to realize a blind equalization algorithm. In Chapter 7 we will present novel concepts of using a radial-basisfunction neural network for reconstructing chaotic dynamics under noisy condition. A specific adaptive training algorithm for realizing the reconstruction task will be presented. Results in terms of the mean-squared-error versus the signal-to-noise ratio will be given and compared with those obtained from conventional reconstruction approaches. Also, as a by-product, a non-coherent detection method used in chaos-based digital communication systems will be realized based on the proposed strategy. In Chapter 8 we will continue our discussion of signal reconstruction techniques. Here, we will discuss the use of a recurrent neural network for reconstructing chaotic dynamics when the signals are transmitted through distorting and noisy channels. A special training algorithm for realizing the reconstruction task will be presented. This problem will also be discussed under the conventional viewpoint of channel equalization. In Chapter 9, the reconstruction of chaotic signals will be considered in terms of chaos vi

Preface

synchronization approaches. In particular, the problem of multiple-access spreadspectrum synchronization for single point to multiple point communication is considered in the light of a chaotic network synchronization scheme, which combines the Pecora-Carroll synchronization scheme and the OOY control method. The results indicate that such a network synchronization scheme is applicable to fast synchronization of multiple chaotic systems, which is an essential condition for realizing single point to multiple point spread-spectrum communications. Simulation results indicate that such a network synchronizationbased communication system is effective and reliable for noisy and multi-path channels. The final chapter summarizes the key results in this research. In closing this preface, we hope that this book has provided a readable exposition

of the related theories for signal reconstructions as well as some detailed descriptions of specific applications in communications. We further hope that the materials covered in this book will serve as useful references and starting points for researchers and graduate students who wish to probe further into the general theory of signal reconstruction and applications. Jiuchao Feng, Guangzhou Chi K. Tse, Hong Kong

vii

Acknowledgements

The completion of this book would have been impossible without the help and support of several people and institutions. We wish to express our most grateful thanks to Prof. Ah-Chung Tsoi from University of Wollongong, Australia, and Prof. Michael Wong from Hong Kong University of Science and Technology, Hong Kong, who have read an early version of this book (which was in the form of a Ph.D. thesis) and have offered countless suggestions and comments for improving the technical contents of the book. We also wish to thank Dr. Francis Lau and Dr. Peter Tam from Hong Kong Polytechnic University, Hong Kong, for many useful and stimulating discussions throughout the course of our research. The first author gratefully acknowledges the Hong Kong Polytechnic University for providing financial support during his Ph.D. study. He also thanks the following institutes and departments for their financial support in the course of preparing this book and in its production: The Natural Science Foundation of Guangdong Province, China (Grant Number 05006506); The Research Foundation of Southwest China Normal University, China (Grant Number 413604); The Natural Science Foundation of Guangdong Province, China, for Team Research (Grant Number 04205783); The Program Foundation for New Century Excellent Talents in Chinese University (Grant Number NCET-04-0813); The Key Project Foundation of the Education Ministry of China (Grant Number 105137); The National Natural Science Foundation of China (Grant Number 60572025); The publishing Foundation for Postgraduate's Textbook in South China University of Technology, China. He is also grateful to his graduate student Shiyuan Wang and Hongjuan Fan for their simulation work in partial chapters. Last, but not least, we wish to thank our families for their patience, understanding and encouragement throughout the years.

To our families Yao Tan and Lang Feng Belinda and Eugene

This page intentionally left blank

Contents

Preface .......................................................................................................... iii Acknowledgements ...................................................................................... ix 1 Chaos and Communications ................................................................... 1 1.1

Historical Account ........................................................................... 2

1.2

Chaos ................................................................................................ 5

1.3

Quantifying Chaotic Behavior ......................................................... 5 1.3.1

Lyapunov Exponents for Continuous-Time Nonlinear

1.3.2

Lyapunov Exponent for Discrete-Time Systems ................. 8

1.3.3

Kolmogorov Entropy ............................................................ 8

1.3.4

Attractor Dimension ........................................................... 10

Systems ................................................................................ 6

1.4

Properties of Chaos ........................................................................ 12

1.5

Chaos-Based Communications ...................................................... 14

1.6

1.7

1.5.1

Conventional Spread Spectrum .......................................... 14

1.5.2

Spread Spectrum with Chaos ............................................. 16

1. 5.3

Chaotic Synchronization .................................................... 16

Communications Using Chaos as Carriers ..................................... 19 1.6.1

Chaotic Masking Modulation ............................................. 19

1.6.2

Dynamical Feedback Modulation ...................................... 20

1.6.3

Inverse System Modulation ................................................ 21

1.6.4

Chaotic Modulation ............................................................ 22

1.6.5

Chaos Shift Keying ............................................................ 22

1.6.6

Differential Chaos Shift Keying Modulation ..................... 24

Remarks on Chaos-Based Communications .................................. 25

Reconstruction of Chaotic Signals with Applications to Chaos-Based Communications

2

1.7.2

Engineering Challenges ........................................... ,.......... 25

Reconstruction of System Dynamics ............................................. 28 2.1.1

Topological Embeddings .................................................... 29

2.1.2

Delay Coordinates .............................................................. 30

2.2

Differentiable Embeddings ............................................................ 33

2.3

Phase Space Reconstruction-Example ........................................ 34

2.4

Problems and Research Approaches .............................................. 39

Fundamentals of Neural Networks ..................................................... .41

3.1

Motivation ...................................................................................... 41

3.2

Benefits of Neural Networks ......................................................... .43

3.3

Radial Basis Function Neural Networks ....................................... .46

3.4

4

Security Issues .................................................................... 25

Reconstruction of Signals ................. ,................................................... 27

2.1

3

1.7.1

3.3.1

Background Theory ........................................................... .46

3.3.2

Research Progress in Radial Basis Function Networks ..... .49

Recurrent Neural Networks ........................................................... 56 3.4.1

Introduction ........................................................................ 56

3.4.2

Topology of the Recurrent Networks ................................. 57

3.4.3

Learning Algorithms .......................................................... 58

Signal Reconstruction in Noisefree and Distortionless Channels ...... 60

4.1

Reconstruction of Attractor for Continuous Time-Varying Systems .......................................................................................... 60

4.2

Reconstruction and Observability .................................................. 62

4.3

Communications Based on Reconstruction Approach ................... 63

4.4

4.3.1

Parameter Estimations ........................................................ 64

4.3.2

Information Retrievals ........................................................ 66

Reconstruction of Attractor for Discrete Time-Varying Systems .......................................................................................... 69

4.5 xii

Summary ........................................................................................ 71

Contents

5

Signal Reconstruction from a Filtering Viewpoint: Theory .... ............ '" 72

5.1

5.2

5.3 6

The Kalman Filter and Extended Kalman Filter ............................ 72 5.1.1 The Kalman Filter .............................................................. 72 5.1.2 Extended Kalman Filter ..................................................... 76 The Unscented Kalman Filter ........................................................ 77 5.2.1 The Unscented Kalman Filtering Algorithm ...................... 78 5.2.2 Convergence Analysis for the UKF Algorithm .................. 82 5.2.3 Computer Simulations ........................................................ 86 Summary ........................................................................................ 89

Signal Reconstruction from a Filtering Viewpoint: Application ............ 91 6.1 Introduction .................................................................................... 91

6.2

Filtering of Noisy Chaotic Signals ................................................. 92 6.2.1 Filtering Algorithm ............................................................ 92 6.2.2 Computer Simulation ......................................................... 94

6.3

Blind Equalization for Fading Channels ...................................... 101 6.3.1 Modeling of Wireless Communication Channels ............. 101 6.3.2 Blind Equalization of Fading Channels with Fixed Channel Coefficients ........................................................ 103 6.3.3 Blind Equalization for Time-Varying Fading Channels .... 106 6.4 Summary ...................................................................................... 109

7

Signal Reconstruction in Noisy Channels .......................................... l1O 7.1 Review of Chaotic Modulation .................................................... 11 0

7.2 7.3

7.4 7.5

F onnulation of Chaotic Modulation and Demodulation .............. 112 On-Line Adaptive Learning Algorithm and Demodulation ......... 116 7.3.1 Description ofthe Network .............................................. 116 7.3.2 Network Growth ............................................................... 118 7.3.3 Network Update with Extended Kalman Filter ................ 119 7.3.4 Pruning of Hidden Units .................................................. 121 7.3.5 Summary of the Flow of Algorithm ................................. 121 Computer Simulation and Evaluation .......................................... 123 Application to Non-coherent Detection in Chaos-Based Communication ............................................................................ 131 xiii

Reconstruction of Chaotic Signals with Applications to Chaos-Based Communications

7.6

Summary ...................................................................................... 139

8

Signal Reconstruction in Noisy Distorted Channels ......................... 140 8.1 Preliminaries ................................................................................ 141 8.1.1 Conventional Equalizers .......................... ,....................... 142 8.1.2 Reconstruction of Chaotic Signals and Equalization ....... 143 8.1.3 Recurrent Neural Network and Equalization ................... 144 8.2 Training Algorithm ...................................................................... 148 8.3 Simulation Study .......................................................................... 151 8.3.1 Chaotic Signal Transmission ............................................ 151 8.3.2 Filtering Effect of Communication Channels ................... 152 8.3.3 Results .............................................................................. 156 8.4 Comparisons and Discussions ...................................................... 161 8.5 Summary ...................................................................................... 164

9

Chaotic Network Synchronization and Its Applications in Communications .................................................................................. 165 9.1

9.2

9.3 10

Chaotic Network Synchronization ............................................... 166 9.1.1 Network Synchronization ................................................. 167 9.1.2 Chaos Contro1. .................................................................. 168 9.1.3 Implementation of the Synchronization Scheme ............. .172 Implementation of Spread-Spectrum Communications ............... 175 9.2.1 Encoding and Decoding ................................................... 175 9.2.2 Test Results for Communications .................................... 178 Summary ...................................................................................... 181

Conclusions ....................................................................................... 183 10.1 Summary of Methods .............................................................. 183 10.2 Further Research ..................................................................... 185

Bibliography ............................................................................................. 188 Index .......................................................................................................... 214 xiv

Chapter 1

Chaos and Communications

Traditionally, signals (encompassing desired signals as well as interfering signals) have been partitioned into two broadly defined classes, i.e., stochastic and deterministic. Stochastic signals are compositions of random waveforms with each component being defined by an underlying probability distribution, whereas deterministic signals are resulted from deterministic dynamical systems which can produce a number of different steady state behaviors including DC, periodic, and chaotic solutions. Deterministic signals may be described mathematically by differential or difference equations, depending on whether they evolve in continuous or discrete-time. DC is a nonoscillatory state. Periodic behavior is the simplest type of steady state oscillatory motion. Sinusoidal signals, which are universally used as carriers in analog and digital communication systems, are periodic solutions of continuous-time deterministic dynamical systems. Deterministic dynamical systems also admit a class of nonperiodic signals, which are characterized by a continuous "noiselike" broad power spectrum. This is called chaos. Historically, at least three achievements were fundamental to the acceptance of communication using chaos as a field worthy of attention and exploitation. The first was the implementation and characterization of several electronic circuits exhibiting chaotic behavior in early 1980's. This brought chaotic systems from mathematical abstraction into application in electronic engineering. The second historical event in the path leading to exploitation for chaosbased communication was the observation make by Pecora and Carroll in 1990 that two chaotic systems can synchronize under suitable coupling or driving conditions. This suggested that chaotic signals could be used for

Historical Account

1.1

communication, where their noise like broadband nature could Improve disturbance rejection as well as security. A third, and fundamental, step was the awareness of the nonlinear (chaos) community that chaotic systems enjoy a mixed deterministic / stochastic nature [1 - 4]. This had been known to mathematicians since at least the early 1970's, and advanced methods from that theory have been recently incorporated in the tools of chaos-based communication engineering. These tools were also of paramount importance in developing the quantitative models needed to design chaotic systems that comply with the usual engineering specifications. The aim of this chapter is to give a brief review of the background theory for chaos-based communications. Based on several dynamical invariants, we will quantitatively describe the chaotic systems, and summarize the fundamental properties of chaos that make it useful in serving as a spreadspectrum carrier for communication applications. Furthermore, chaotic synchronization makes it possible for chaos-based communication using the conventional coherent approach. In the remaining part of this chapter, several fundamental chaotic synchronization schemes, and several chaosbased communication schemes will be reviewed. Finally, some open issues for chaos-based communications will be discussed.

1.1

Historical Account

In 1831, Faraday studied shallow water waves

III

a container vibrating

vertically with a given frequency OJ. In the experiment, he observed the sudden appearance of sub harmonic motion at half the vibrating frequency (OJ / 2) under certain conditions. This experiment was later repeated by Lord

Rayleigh who discussed this experiment in the classic paper Theory of Sound, published in 1877. This experiment has been repeatedly studied since

1960's. The reason why researchers have returned to this experiment is that the sudden appearance of sub harmonic motion often prophesies the prelude to chaos. 2

Chapter 1

Chaos and Commnnications

Poincare discovered what is today known as homo clinic trajectories in the state space. In 1892, this was published in his three-volume work on Celestial Mechanics. Only in 1962 did Smale prove that Poincare's homoclinic trajectories are chaotic limit sets [5]. In 1927, Van der Pol and Van der Mark studied the behavior of a neon bulb RC oscillator driven by a sinusoidal voltage source [6]. They discovered that by increasing the capacitance in the circuit, sudden jumps from the drive frequency, say w to w/2, then to w/3, etc., occurred in the response. These frequency jumps were observed, or more accurately heard, with a telephone receiver. They found that this process of frequency demultiplication (as they called it) eventually led to irregular noise. In fact, what they observed, in today's language, turned out to be caused by bifurcations and chaos. In 1944, Levinson conjectured that Birkhoffs remarkable curves might occur in the behavior of some third-order systems. This conjecture was answered affirmatively in 1949 by Levinson [7]. Birkhoff proved his famous Ergodic Theorem in 1931 [8]. He also discovered what he termed remarkable curves or thick curves, which were also studied by Charpentier in 1935 [9]. Later, these turned out to be a chaotic attractor of a discrete-time system. These curves have also been found to be fractal with dimension between 1 and 2. In 1936, Chaundy and Phillips [10] studied the convergence of sequences defined by quadratic recurrence formulae. Essentially, they investigated the logistic map. They introduced the terminology that a sequence oscillates irrationally. Today this is known as chaotic oscillation. Inspired by the discovery made by Van der Pol and Van der Mark, two mathematicians, Cartwright and Littlewood [11] embarked on a theoretical study of the system studied earlier by Van der Pol and Van der Mark. In 1945, they published a proof of the result that the driven Van der Pol system can exhibit nonperiodic solutions. Later, Levinson [7] referred to these solutions as singular behavior. Melnikov [12] introduced his perturbation method for chaotic systems in 1963. This method is mainly applied to driven dynamical systems. 3

1.1

Historical Accouut

In 1963, Lorenz [13], a meteorologist, studied a simplified model for thermal convection numerically. The model (today called the Lorenz model) consisted of a completely deterministic system of three nonlinearly coupled ordinary differential equations. He discovered that this simple deterministic system exhibited irregular fluctuations in its response without any external element of randomness being introduced into the system. Cook and Roberts [14] discovered chaotic behavior exhibited by the Rikitake two-disc dynamo system in 1970. This is a model for the time evolution of the earth magnetic field. In 1971, Ruelle and Takens [15] introduced the term strange attractor for dissipative dynamical systems. They also proposed a new mechanism for the on-set of turbulence in the dynamics offluids. It was in 1975 that chaos was formally defined for one-dimensional

transformations by Li and Yorke [16]. They went further and presented sufficient conditions for so-called Li -Yorke chaos to be exhibited by a certain class of one-dimensional mappings. In 1976, May called attention to the very complicated behavior which included period-doubling bifurcations and chaos exhibited by some very simple population models [17]. In 1978, Feigenbaum discovered scaling properties and universal constants (Feigenhaum's number) in one-dimensional mappings [18]. Thereafter, the idea of a renormalization group was introduced for studying chaotic systems. In 1980, Packard et al. [19] introduced the technique of state-space reconstruction using the so-called delay coordinates. This technique was later placed on a firm mathematical foundation by Takens. In 1983, Chua [20] discovered a simple electronic circuit for synthesizing the specific third-order piecewise-linear ordinary differential equations. This circuit became known as Chua's circuit (see the following chapter). What makes this circuit so remarkable is that its dynamical equations have been proven to exhibit chaos in a rigorous sense. Ott, Grebogi and Yorke, in 1990 [21], presented a method for controlling unstable trajectories embedded in a chaotic attractor. At the same time, there was another course of events leading to the field of chaos. This was the study of nonintegrable Hamiltonian systems in classical 4

Chapter 1

Chaos and Commnnications

mechanics. Research in this field has led to the formulation and proof of the Kolmogorov-Amold-Moser (KAM) theorem in the early 1960's. Numerical studies have shown that when the conditions stated by the KAM theorem fails, then stochastic behavior is exhibited by nonintegrable Hamiltonian systems. Remarks: Today, chaos has been discovered in bio-systems, meteorology, cosmology, economics, population dynamics, chemistry, physics, mechanical and electrical engineering, and many other areas. The research direction has been transferring from fmding the evidence of chaos existence into applications and deep theoretical study.

1.2

Chaos

There are many possible definitions of chaos for dynamical systems, among which Devaney's definition (for discrete-time systems) is a very popular one because it applies to a large number of important examples.

Theorem 1.1 [22] Let Q be a set.f: Q---+ Q is said to be chaotic on Q if: (1) fhas sensitive dependence on initial conditions, i.e., there exists

(» 0

such that, for any x EQ and any neighborhood U of x, there exists Y E U and

n?oO such that IF(x)-F(Y)1 >0; (2) f is topological transitive, i.e., for any pair of open sets V, We Q there exists k> 0 such that fk (V) n

w

-:j:.

0;

(3) periodic points are dense in Q.

1.3

Quantifying Chaotic Behavior

Lyapunov exponents, entropy and dimensionality are usually used to quantify (characterize) a chaotic attractor's behavior. Lyapunov exponents indicate the average rates of convergence and divergence of a chaotic atiractor in the state space. Kolmogorov Entropy (KE) is used to reveal the rate of information 5

1.3

Quantifying Chaotic Behavior

loss along the attractor. Dimensionality is used to quantify the geometric structure of a chaotic attractor.

1.3.1 Lyapunov Exponents for Continuous-Time Nonlinear Systems The determination of Lyapunov exponents is important in the analysis of a possibly chaotic system, since Lyapunov exponents not only show qualitatively the sensitive dependence on initial conditions but also give a quantitative measure of the average rate of separation or attraction of nearby trajectories on the attractor. Here, we only state the definitions of Lyapunov exponents for continuous-time and discrete-time (see next subsection) nonlinear systems. More detailed descriptions of various algorithms for calculating Lyapunov exponents can be found in references [23 - 33]. In the direction of stretching, two nearby trajectories diverge; while in the directions of squeezing, nearby trajectories converge. Ifwe approximate this divergence and convergence by exponential functions, the rates of stretching and squeezing would be quantified by the exponents. These are called the Lyapunov exponents. Since the exponents vary over the state space, one has to take the long time average exponential rates of divergence (or convergence) of nearby orbits. The total number of Lyapunov exponents is equal to the degree of freedom of the system. If the system trajectories have at least one positive Lyapunov exponent, then those trajectories are either unstable or chaotic. If the trajectories are bounded and have positive Lyapunov exponents, the system definitely includes chaotic behavior. The larger the positive exponent is, the shorter the predictable time scale of system. The estimation of the largest exponent therefore assumes a special importance. For a given continuous-time dynamical system in an m-dimensional phase space, we monitor the long-term evolution of an infinitesimal m-sphere of initial conditions. The sphere will evolve into an m-ellipsoid due to the locally

6

Chapter 1

Chaos and Communications

deforming nature of the flow. The ith one-dimensional Lyapunov exponent can be defined in terms of the length of the ellipsoidal principal axis Ii (I):

A;

=

lim!lnI/JI) I, Hoo 1 1;(0)

(1.1)

whenever the limit exists [24,25]. Thus, the Lyapunov exponents are related to the expanding or the contracting nature of the principal axes in phase space. A positive Lyapunov exponent describes an exponentially increasing separation of nearby trajectories in a certain direction. This property, in tum, leads to the sensitive dependence of the dynamics on the initial conditions, which is a necessary condition for chaotic behavior. Since the orientation of the ellipsoid changes continuously as it evolves, the directions associated with a given exponent vary in a complicated way through the attractor. We cannot therefore speak of a well defined direction associated with a given exponent. For systems whose equations of motion are known explicitly, Benettin el

al. [25] have proposed a straightforward technique for computing the complete Lyapunov spectrum. This method can be described in principle as follows. Let an m-dimensional compact manifold M

be the state space of a

dynamical system. The system on M is a nonlinear differentiable map ([J:

M ---+M, which can be conveniently described by the following difference equation: x(n)

= ([J(x(n

-1»

= ([In (x(o».

(1.2)

Let v(O) denote an initial perturbation of a generic point x(O), and & be a sufficiently small constant. Consider the separation of trajectories of the unperturbed and perturbed points after n iterations:

II ([In (x(O»

- ([In (x(O) + &v(O»

a

II = II D([Jn (x(O»v(O)& II 0(&2) = II(

D([Jn (x(k» }(O)&II + 0(&2),

(1.3) 7

1.3

Quautifying Chaotic Behavior

where D0+ N -->00 NT n=O

(1.7)

The order in which the limits in the above expression are taken is immaterial. The limit s ----> 0 makes KE independent of the particular partition. The main properties ofKE are as follows: (1) The entropy K (averaged) determines the rate of change in information entropy (i.e., Eq. (1.6)) as a result of a purely dynamical process of mixing of trajectories in phase space. (2) The entropy is a metric invariant of the system, i.e., its value being independent of the way that the phase space is divided into cells and coarsened. (3) Systems with identical values of entropy are in a certain sense isomorphic to each other [39, 40], i.e., these systems must have identical statistical laws of motion. (4) When applied to prediction, KE can be interpreted as the average rate at which the accuracy of a prediction decays as prediction time increases, i.e., the rate at which predictability will be lost [37].

9

1.3

Quantifying Chaotic Behavior

1.3.4

Attractor Dimension

Long-term chaotic motion in dissipative dynamical systems is confined to a strange attractor whose geometric structure is invariant to the evolution of the dynamics. Typically, a strange attractor is a fractal object and, consequently, there are many possible notions of the dimension for strange attractor. Here, we discuss some well-known and widely accepted definitions of attractor dimension. We also discuss the simple relationships to Lyapunov exponents and entropy. Dissipative chaotic systems are typically ergodic. All initial conditions within the system's basin of attraction lead to a chaotic attractor in the state space, which can be associated with a time-invariant probability measure. Intuitively, the dimension of the chaotic attractor should reflect the amount of information required to specify a location on the attractor with a certain precision. This intuition is formalized by defining the information dimension, d" of the chaotic attractor as -1· In p[Bx (&)] d[ - 1m , 0 In&

(1.8)

where p[Bx(&)] denotes the mass of the measure p contained in a ball of radius

&

centered at the point x in the state space [2, 41]. Information dimen-

sion is important from an experimental viewpoint because it is straightforward to estimate. The mass, p[Bx(&)] ' can be estimated by (1.9) where U(-) is the Heaviside function, and M is the number of points in the phase space. In typical experiments, the state vector x is estimated from a delay embedding of an observed time-series [42]. The information dimension as defmed above, however, depends on the particular point x in the state space being considered. Grassberger and Procaccia's approach eliminates this dependence [43] by defming the quantity 10

Chapter 1 Chaos and Communications

(1.10) and then defining the correlation dimension de as de = lim In C(l'). B--+O

In practice, one usually plots In C( £

)

(1.11 )

Inl'

as a function of In £ and then measures

the slope of the curve to obtain an estimate of de [44 - 66]. It is often the case that d, and de are approximately equal. The box-counting dimension (capacity), do, of a set is defined as -1· 1n.ll/(c) d0 - 1m &-->0 1n(l/ c)

(1.12)

,

where .11/(.0) is the minimum number of N-dimensional cube of side length £

needed to cover the set. There is a meaningful relationship between the Lyapunov dimension and

Lyapunov exponents for chaotic systems [41, 67, 68]. If AI, ... , AN are the Lyapunov exponents of a chaotic system, then the Lyapunov dimension, dL , is defined as (1.13) where k

=

max {i : ｾ＠ + ... + Ai > O}. It has been shown that do ?cd"

､Ｌｾ｣＠

[43, 69].

Equation (1.14) suggests that only the first k + 1 Lyapunov exponents are important for specifying the dimensionality of the chaotic attractor. Kaplan et

al. [67, 68] conjectured that d, = dL in "almost" all cases. Clearly, if this is correct, then Eq. (1.13) provides a straightforward way to estimate the attractor dimension when the dynamical equations of motion are known. The relation between KE and Lyapunov exponents is also available. In one-dimensional maps, KE is just the Lyapunov exponent [70]. In higherdimensional systems, we lose information about the system because the cell in which it was previously located spreads over new cells in phase space at a 11

1.4

Properties of Chaos

rate determined by the Lyapunov exponents. The rate K at which the information about the system is lost is equal to the (averaged) sum of the positive Lyapunov exponents [71], as shown by (1.14) where A's are the positive Lyapunov exponents of the dynamical system being considered.

1.4

Properties of Chaos

It is now well-known that a deterministic dynamical system is one whose

state evolves with time according to a deterministic evolution rule. The time evolution of the state is completely determined by the initial state of the system, the input, and the rule. For example, the state of a digital filter is determined by the initial state of the filter, the input, and a difference equation which describes the evolution of the state from one time instant to the next. In contrast to a stochastic dynamical system, which may follow any number of different trajectories from a given state according to some probabilistic rule, trajectories of a deterministic dynamical system are unique. From any given state, there is only one "next" state. Therefore, the same system started twice from the same initial state with the same input will follow precisely the same trajectory through the state space. Deterministic dynamical systems can produce a variety of steady-state behaviors, the most familiar of which are stationary, periodic, and quasi-periodic solutions. These solutions are "predictable" in the sense that a small piece of a trajectory enables one to predict the future behavior along that trajectory. Chaos refers to solutions of deterministic dynamical systems which, while predictable in the short-term, exhibit long-term unpredictability. Since the initial state, input, and rule uniquely determine the behavior of a deterministic dynamical system, it is not obvious that any "unpredictability" 12

Chapter 1

Chaos and Communications

is possible. Long-term unpredictability arises because the dynamics of a chaotic system persistently amplifies errors in specifying the state. Thus, two trajectories starting from nearby initial conditions quickly become uncorrelated. This is because in a chaotic system, the precision with which the initial conditions must be specified in order to predict the behavior over some specified time interval grows exponentially with the length of the prediction interval. As a result, long-term prediction becomes impossible. This long-term unpredictability manifests itself in the frequency domain as a continuous power spectrum, and in the time domain as random "noiselike" signal. To get a better idea of what chaos is, here is a list of its main characteristics (properties) : (1) Chaos results from a deterministic nonlinear process. (2) The motion looks disorganized and erratic, although sustained. In fact, it can usually pass all statistical tests for randorrmess (thereby we cannot distinguish chaotic data from random data easily), and has an invariant probability distribution. The Fourier spectrum (power spectrum) is "broad" (noiselike) but with some periodicities sticking up here and there [72, 73]. (3) Details of the chaotic behavior are hypersensitive to changes in initial conditions (minor changes in the starting values of the variables). Equivalently, chaotic signals rapidly decorrelate with themselves. The autocorrelation function of a chaotic signal has a large peak at zero and decays rapidly. (4) It can result from relatively simple systems. In nonautonomous system, chaos can take place even when the system has only one state variable. In autonomous systems, it can happen with as few as three state variables. (5) For given conditions or control parameters, chaos is entirely selfgenerated. In other words, changes in other (i.e., external) variables or parameters are not necessary. (6) It is not the result of data inaccuracies, such as sampling error or measurement error. Any specific initial conditions (right or wrong), as long as the control parameter is within an appropriate range, can lead to chaos. (7) In spite of its disjointed appearance, chaos includes one or more types 13

1.5

Chaos-Based Commuuicatious

of order or structure. The phase space trajectory may have fractal property (self-similarity). (8) The ranges of the variables have finite bounds, which restrict the attractor to a certain finite region in the phase space. (9) Forecasts of long-term behavior are meaningless. The reasons are sensitivity to initial conditions and the impossibility of measuring a variable to absolute accuracy. Short-term predictions, however, can be relatively accurate. (10) As a control parameter changes systematically, an initially nonchaotic system follows one of a few typical scenarios, called routes to chaos.

1.5 1.5.1

Chaos-Based Communications Conventional Spread Spectrum

In recent years, there has been explosive growth in personal communications, the aim of which is to guarantee the availability of voice and/or data services between mobile communication terminals. In order to provide these services, radio links are required for a large number of compact terminals in densely populated areas. As a result, there is a need to provide highfrequency, low-power, low-voltage circuitry. The huge demand for communications results in a large number of users; therefore, today's communication systems are limited primarily by interference from other users. In some applications, the efficient use of available bandwidth is extremely important, but in other applications, where the exploitation of communication channels is relatively low, a wideband communication technique having limited bandwidth effciency can also be used. Often, many users must be provided with simultaneous access to the same or neighboring frequency bands. The optimum strategy in this situation, where every user appears as interference to every other user, is for each communicator's signal to look like white noise which is as wideband as possible. 14

Chapter 1

Chaos and Communications

There are two ways in which a communicator's signal can be made to appear like wideband noise: (1) spreading each symbol using a pseudo-random sequence to increase the bandwidth of the transmitted signal; (2) representing each symbol by a piece of "noiselike" waveform [74]. The conventional solution to this problem is the first approach: to use a synchronizable pseudo-random sequence to distribute the energy of the information signal over a much larger bandwidth to transmit the baseband information. The transmitted signal appears similar to noise and is therefore diffcult to detect by eavesdroppers. In addition, the signal is difficult to jam because its power spectral density is low. By using orthogonal spreading sequences, multiple users may communicate simultaneously on the same channel, which is termed Direct Sequence Code Division Multiple Access (DS/CDMA). Therefore, the conventional solution can: (1) combat the effects of interference due to jamming, other users, and multipath effects; (2) hide a signal "in the noise" by transmitting it at low power; and (3) have some message privacy in the presence of eavesdroppers. With rapidly increasing requirements for some new communication services, such as wideband data and video, which are much more spectrumintensive than voice service, communication networks are already reaching their available resource limitation. Some intrinsic shortcomings of the convenient DS/CDMA have been known. For example, the periodic nature of the spreading sequences, the limited number of available orthogonal sequences, and the periodic nature of the carrier, are imposed to DS/CDMA systems in order to achieve and maintain carrier and symbol synchronization. One further problem is that the orthogonality of the spreading sequences requires the synchronization of all spreading sequences used in the same frequency band, i.e., the whole system must be synchronized. Due to different propagation times for different users, perfect synchronization can never be achieved in real systems [75]. In addition, DS/CDMA systems using binary spreading sequences do not provide much protection against 15

1.5

Chaos-Based Communications

two particular interception methods: the carrier regeneration and the code clock regeneration detectors [76]. This is due to the binary nature of the spreading sequences used in binary waveforms. The intrinsic properties of chaotic signals stated previously provide an alternative approach to making a transmission "noiselike". Specifically, the transmitted symbols are not represented as weighted sums of periodic basis functions but as inherently nonperiodic chaotic signals, which will be described in the following subsections.

1.5.2

Spread Spectrum with Chaos

The properties of chaotic signals resemble in many ways those of the stochastic ones. Chaotic signals also possess a deterministic nature, which makes it possible to generate "noiselike" chaotic signals in a theoretically reproducible manner. Therefore, a pseudo-random sequence generator is a "practical" case of a chaotic system, the principal difference being that the chaotic system has an infinite number of (analog) states, while pseudo-random generator has a finite number (of digital states). A pseudo-random sequence is produced by visiting each state of the system once in a deterministic manner. With only a finite number of states to visit, the output sequence is necessarily periodic. By contrast, an analog chaos generator can visit an infinite number of states in a deterministic manner and therefore produces an output sequence, which never repeats itself. With appropriate modulation and demodulation techniques, the "random" nature and "noiselike" spectral properties of chaotic electronic circuits can be used to provide simultaneous spreading and modulation of a transmission.

1.5.3

Chaotic Synchronization

How then would one use a chaotic signal in communication? A first approach would be to hide a message in a chaotic carrier and then extract it 16

Chapter 1

Chaos and Communications

by some nonlinear, dynamical means at the receiver. If we do this in realtime, we immediately lead to the requirement that somehow the receiver must have a duplicate of the transmitter's chaotic signal or, better yet, synchronize with the transmitter. In fact, synchronization is a requirement of many types of communication, not only chaotic possibilities. Early work on synchronous coupled chaotic systems was done by Yamada and Fujisaka [77, 78]. In that work, some sense of how the dynamics might change was brought out by a study of the Lyapunov exponents of synchronized coupled systems. Later, Afraimovich et al. [79] exposed many of the concepts necessary for analyzing synchronous chaos. A crucial progress was made by Pecora and Carroll [80 - 84], who have shown theoretically and experimentally that two chaotic systems can be synchronized. This discovery bridges between chaos theories and communications, and opens up a new research area in communications using chaos. The driving response synchronization configuration proposed by Pecora and Carroll is shown in Fig. 1.1, in which the Lorenz system is used in the transmitter and the receiver, where Xi or X; (i = 1,2,3, r standing for the response system) is the state variable of the Lorenz system [13], Ii is the ith state equation, and 11 (t) is additive channel noise. The drive-response synchronization method indicates that if a chaotic system can be decomposed into subsystems, a drive system system

(X2' X3)

Xl

and a conditionally stable response

in this example [82], then the identical chaotic system at the

receiver can be synchronized when driven with a common signal. The output

x; and x; will follow the signals

signals

X2

and

X3.

For more discussions on

chaotic synchronization, see [83].

ｸｊＨｉＩｾＱＳｘ＠

ｾｩＺｬＨｴＩｘＬ＠

X2(t), x3(t)) ｾｩ｣ｴＩｬｸＨＬ＠

x 2 (t), x 3(t)) 1(I),

x 1(t)

y(l)

xf ＨｴＩｾｬ＠ ｸＲＨｴＩｾＱ＠

xj ＨｴＩｾＱＳ＠

xit), x 3(t))

(Y (f). ｸｾＨｴＩＬ＠

x3(t)) Y (I), x2(t), x{(I))

Y (I), ｸｾＨｴＩＬ＠

x;(t)

xl(tll

11(1)

Drive system

Response system

Figure 1.1 Drive-response synchronization schematic diagram, in which

Xi

or

x;

(i = 1,2,3, r stands for the response system) is the state variable of the Lorenz system [13], Ii is the ith state equation, and 17 (t) is additive channel noise 17

1.5

Chaos-Based Communications

Based on the self-synchronization properties of chaotic systems, some chaotic communication systems using chaotic carriers have been proposed. Since the performance of such communication systems will strongly depend on the synchronization capability of chaotic systems, the robustness of se1fsynchronization in the presence of white noise needs to be explored [85]. Inspired by Pecora and Carroll's work, many other synchronization schemes have been proposed, including error feedback synchronization [87], inverse system synchronization [88], adaptive synchronization [89], generalized synchronization [90], etc. The error feedback synchronization is borrowed from the field of automatic control. An error signal is derived from the difference between the output of the receiver system and that received from the transmitter. The error signal is then used to modify the state of the receiver such that it can be synchronized with the transmitter. The operating theory of the inverse system synchronization scheme is as follows. If a system L with state x(t) and input set) produces an output yet), 1

then its inverse L- produces an output Yr(t) = s(t) and its state xr(t) has synchronized with x(t). Adaptive synchronization scheme makes use of the procedure of adaptive control and introduces the time dependent changes in a set of available system parameters. This scheme is realized by perturbing the system parameters whose increments depend on the deviations of the system variables from the desired values and also on the deviations of the parameters from their correct values corresponding to the desired state. Generalized synchronization of the uni-directionally coupled systems

x = F(x)

(x

y = H(y, x)

E

ffi.", drive)

(y

E

ffi.", response)

(1.15) (1.16)

occurs for the attractor Ax c lR of the drive system if an attracting synchronization set

18

Chapter 1 Chaos and Commnnications

M = {(X,y) E A/]Rm : y = H(x)} exists and is given by some function ｈＺａｲｾｹ｣｝ｒｭＮ＠

(1.17) Also, M possesses an

open basin 13 ::J M such that: lim II y(t) - H(x(t)) 11= 0,

\t(x(O), y(O))

E

13.

1-'>00

(1.18)

It was reported in [91] that for a linear bandpass communication channel with

additive white Gaussian noise (AWGN), drive-response synchronization is not robust (signal-to-noise ratio, > 30 dB is required) and the continuoustime analog inverse system exhibits extreme noise sensitivity (SNR > 40 dB is required to maintain synchronization). Further, recent studies of chaotic synchronization, where significant noise and filtering have been introduced to the channel, indicate that the performance of chaotic synchronization schemes is worse, at low SNR, than that of the best synchronization schemes for sinusoids [92].

1.6

Communications Using Chaos as Carriers

The use of modulating an aperiodic or nonperiodic chaotic waveform, instead of a periodic sinusoidal signal, for carrying information messages has been proposed since chaotic synchronization phenomenon was discovered. In particular, chaotic masking [85,93], dynamical feedback modulation [94], inverse system modulation [95], chaotic modulation [88, 96 - 10 1], chaoticshift-keying (CSK) [102 - 112], and differential chaos shift keying (DCSK) [113], have been proposed. In the following, we will provide a brief summary of these schemes.

1.6.1

Chaotic Masking Modulation

The basic idea of a chaotic masking modulation scheme is based on chaotic signal masking and recovery. As shown in Fig. 1.2, in which the Lorenz 19

1.6

Communications Using Chaos as Carriers

system is also used as the chaotic generator, we add a noiselike chaotic signal

Xl (t)

to the information signal met) at the transmitter, and at the

receiver the masking signal is removed [85, 86,93]. The received signaly(t), consisting of masking, information and noise signals, is used to regenerate the masking signal at the receiver. The masking signal is then subtracted from the received signal to retrieve the information signal denoted by

m(t).

The regeneration of the masking signal can be done by synchronizing the receiver chaotic system with the transmitter chaotic system. This communication system could be used for analog and digital information data. Cuomo et al. [85] built a Lorenz system circuit and demonstrated the performance

of chaotic masking modulation system with a segment of speech from a sentence. The performance of the communication system greatly relies on the synchronization ability of chaotic systems. The masking scheme works only when the amplitudes of the information signals are much smaller than the masking chaotic signals. .i:[ (1)=/[ (JJ (I), ｸｾＨｴＩＬ＠

.i:[(I)=I[(x[(t), x 2(1), "3(1» xil)=Iz{x[(I), ﾷｩＺＳＨｴＩ］ｬｾ｛ｉＬ＠

x2(1), x3(1»

xj(l))

.i:2(t)=iz( Y (f), x;[(t), x 3(1))

,iej(t)=i 3( Y (f), x 2(t), x 3(1»

x 2(1), x3(1» l1(t)

Drive system

in (I)

Response system

Figure 1.2 Block diagram of a chaotic masking modulation communication system, in which

Xi

or

X; (i = 1,2, 3, r stands for the response system) is the state variable

of the Lorenz system [13], Ii is the ith state equation, 77 (t) is additive channel noise, met) and

1.6.2

m(t) are the injected message signal and the recovered message signal

Dynamical Feedback Modulation

To avoid the restriction of the small amplitude of the information signal, another modulation scheme, called dynamical feedback modulation, has been proposed in [94]. As shown in Fig. 1.3, in which the Lorenz system is used again as the chaotic generator, the basic idea is to feedback the information signal into the chaotic transmitter in order to have identical 20

Chapter 1

Chaos and Communications

input signals for both the chaotic transmitter and the receiver. Specifically, the transmitted signal, consisting of the information signal met) and the chaotic signal Xl (t), is communicated to the receiver which is identical to the chaotic transmitter. Since the reconstructed signal

x; (t) will be identical to

x(t) in the absence of noise '7(t), the information signal met) can be decoded from the received signal. m(t) ｾｩＺＯｴＩｬＨｸＫｭＬ＠

xit), X3(t» xit), x3(t)) x 2(t), x3(t» ｸＲＨｴＩｾｬｩＫｭＬ＠ ｸＳＨｴＩｾｬｩＫｭＬ＠

ｸｬＨｴＩｾ＠

(y(t), -'2(t), x](t)) xW), xj(t)) y(t), x2(t), xj(t) ｸｍｾｬｩｹＨｴＩＬ＠ ｸｗＩｾＱＳＨ＠

met) Drive system

Response system

Figure 1.3 Block diagram of a dynamical feedback modulation communication system, in which

Xi

or X; (i = 1, 2, 3, r stands for the response system) is the state

variable of the Lorenz system [13], (is the ith state equation, noise, met) and

1]

(t) is additive channel

m(t) are the injected message signal and the recovered message signal

This analog communication technique can be applied to binary data communication by setting met) = C if the binary information data is one, and met) = - C if the binary data is zero. Since the feedback information will

affect the chaotic property, the information level C should be scaled carefully to make the transmitter chaotic to maintain the desired communication security.

1.6.3

Inverse System Modulation

In the inverse system approach [95], the transmitter consists of a chaotic system which is excited by the information signal set). The output yet) of the transmitter is chaotic. The receiver is simply the inverse system, i.e., a system which produces ret) = set) as output when excited by yet) and started from the same initial condition. If the system is properly designed, the output ret) will approach set), regardless of the initial conditions. 21

1.6

Communications Using Chaos as Carriers

1.6.4

Chaotic Modulation

In chaotic modulation [88,96-101], the message signal is injected into a chaotic system as a bifurcation "parameter,,(j), with the range of the bifurcation parameter chosen to guarantee motion in a chaotic region (for more details, see Sec. 7.2). The main advantage of the chaotic modulation scheme is that it does not require any code synchronization, which is necessary in traditional spread spectrum communication systems using coherent demodulation techniques. The crucial design factor is, however, the retrieval of the bifurcation "parameter" variation from the receiving spread spectrum signal, which may be distorted by nonideal channel and contaminated by noise (one of the goals of this book is to investigate signal reconstruction techniques at the receiving end such that the bifurcation parameter and hence the injected message can be recovered).

1.6.5

Chaos Shift Keying

In binary CKS [102-112] as shown in Fig. 1.4 (a), an information signal is encoded by transmitting one chaotic signal

Zj (t)

for a "1" and another

chaotic signal zo(t) to represent the binary symbol "0". The two chaotic signals come from two different systems (or the same system with different parameters); these signals are chosen to have similar statistical properties. Two demodulation schemes are possible: coherent and non-coherent. The coherent receiver contains copies of the systems corresponding to "0" and "1". Depending on the transmitted signal, one of these copies will synchronize with the incoming signal and the other will desynchronize at the receiver. Thus, one may determine which bit is being transmitted. A coherent demodulator is shown in Fig. 1.4 (b), in which Zt(t) and zo(t) are the regenerated chaotic signals at the receiver.

(j)

Bifurcation parameters determine the dynamical behavior of a dynamical system. For some selected range of the parameter values, the system can demonstrate chaotic behavior [22].

22

Chapter 1

Chaos and Communications

Transmitter x(t)

t

"0" Digital infomlation to be transmitted

I I

: - - - - ________________ Receiver I I ______ Channel ｾｌ＠ _ (al Correlator

yet) - - , - - - - - - - - - - 1 Synchronization circuit Symbol duration

Synchronization circuit

Correlator

Digital infomlation received Threshold detector (b)

Figure 1.4 Chaos shift keying digital communication system. Block diagrams of (a) the system, and (b) a coherent CSK demodulator

One type of non-coherent receivers requires the transmitted chaotic signals having different bit energies for "1" and "0". By comparing the bit energy with a decision threshold, one can retrieve the transmitted source information signal. Moreover, other non-coherent schemes exploit the distinguishable property of the chaotic attractors for demodulation, such as in Tse et al. [114]. In particular, if the two chaotic signals come from the same system

23

1.6

Communications Using Chaos as Carriers

with different bifurcation parameters, demodulation can be performed by estimating the bifurcation parameter of the "reconstructed" chaotic signals.

1.6.6

Differential Chaos Shift Keying Modulation

When the channel condition is so poor that it is impossible to achieve chaotic synchronization, a differential chaotic modulation technique for digital communication, called DCSK, has been introduced [113]. This modulation scheme is similar to that of the differential phase shift keying (DPSK) in the conventional digital communication except that the transmitted signal is chaotic. That is, in DCSK, every symbol to be transmitted is represented by two sample functions. For bit" 1", the same chaotic signal are transmitted twice in succession while bit "0" is sent by transmitting the reference chaotic signal followed by an inverted copy of the same signal. At the receiver the two received signals are correlated and the decision is made by a zero-threshold comparator. The DCSK technique offers additional advantages over the CSK: (l) The noise performance of a DSCK communication system in terms of

bit error rate (BER) versus EblNo (Eb is the energy per bit and No is the power spectral density of the noise introduced in the channel) outperforms the BER of a standard non-coherent CSK system. For sufficiently large bit duration, the noise performance of DCSK is comparable to that of a conventional sinusoid-based modulation scheme. In particular, EblNo = 13.5 dB is required for BER= 10-3 [115]. (2) Because synchronization is not required, a DCSK receiver can be implemented using very simple circuitry. (3) DCSK is not as sensitive to channel distortion as coherent methods since both the reference and the information-bearing signal pass through the same channel. The main disadvantage of DCSK results from differential coding: Eb is doubled and the symbol rate is halved. 24

Chapter 1

1.7 1.7.1

Chaos and Communications

Remarks on Chaos-Based Communications Security Issues

Recent studies [116 - 118] have shown that communication schemes using chaotic or hyperchaotic sources have limited security. Therefore, most of the chaos-based communication schemes are based on the viewpoint that security is an added feature in a communication system, which may be implemented by adding encryption/decryption hardware at each end of the system.

1. 7.2

Engineering Challenges

The field of "communications with chaos" presents many challenging research and development problems at the basic, strategic, and applied levels. The building blocks with which to construct a practical chaos-based spread spectrum communication system already exist: chaos generators, modulation schemes, and demodulators. Nevertheless, further research and development are required in all of these subsystems in order to improve robustness to a level that can be comparable to existing conventional system. Synchronization schemes for chaotic spreading signals are not yet sufficiently robust to be competitive with pseudo-random spreading sequences. Nevertheless, they do offer theoretical advantages in terms of basic security level. Furthermore, an analog implementation of chaotic spreading may permit the use of simple low power, high-frequency circuitry. Although an improved scheme, called frequency modulation DCSK (FMDCSK) [115], shows a better performance under multipath environment, channel characteristics are not fully taken into account yet, which limits its realizability in practical environments. Finally, there are still many practical problems that need to be solved, for example, the extension of multiple access design is a practical challenging 25

1.7

Remarks on Chaos-Based Communications

issue involving both system level and basic research. The effects of bandwidth limitation also presents different problems to the practical imple-mentation of such systems. In summary, chaos provides a promising approach for communications. It should be emphasized here that the field of chaos communications is very young: much fundamental work as well as practical problems need to be addressed before high-performance robust chaos-based communication systems can be generally available.

26

Chapter 2

Reconstruction of Signals

In most experimental situations we have to measure some variables from the system under consideration. In many cases these variables are often converted to electrical signals, which can be easily measured by precision instrumentation and later processed by computers. Electrical sensors are widely used to measure temperature, velocity, acceleration, light intensity, activity of human organs, etc. Modem techniques of data acquisition measure the signals in a specific way-they are sampled both in time and space (AID conversion, finite word-length effects, quantization, roundoff, overflow), making it possible to store and process the data using computers. When signals are collected from an experimental setup, several questions are often asked. What kind of information do the measured signals reflect about the system? How do we know if the signals are of sufficient integrity to represent the system in view of possible distortions and noise contamination? What conclusions can be drawn about the nature of the system and its dynamics? These are the basic questions to ask when we attempt to derive useful information from the measured signals. Before we can answer these questions, an important process to be considered is a faithful reconstruction of the signals. In this chapter, we consider the theoretical foundation of signal reconstruction and illustrate how signals can be reconstructed in practice even though the signals may appear "disordered" or "random-like", such as those produced from chaotic systems.

2.1

Reconstruction of System Dynamics

2.1

Reconstruction of System Dynamics

It is seldom the case that all relevant dynamical variables can be measured

in an experiment. Suppose some of the variables are available from measurements. How can we proceed to study the dynamics in such a situation? A key element in solving this general class of problems is provided by the embedding theory [1]. In typical situations, points on a dynamical attractor in the full system phase space have a one-to-one correspondence with measurements of a limited number of variables. This fact is useful for signal reconstruction. By definition, a point in the state space carries complete information about the current state of the system. If the equations defining the system dynamics are not explicitly known, this phase space is not directly accessible to the observer. A one-to-one correspondence means that the state space can be identified by measurements. Assume that we can simultaneously measure m variables YI(t), Y2(t),'" , Ym(t), which can be denoted by the vector y. This m-dimensional vector can

be viewed as a function of the system state x(t) in the full system phase space: y = F(x) =[J;(x) h(x) ... fm(x)]T.

(2.1)

We call the function F the measurement function and the m-dimensional vector space in which the vector y lies the reconstruction space. We have grouped the measurements as a vector-valued function F of x. The fact that F is a function is a consequence of the definition of the state. Specifically, since information about the system is determined uniquely by the state, each measurement is a well-defined function of x. As long as m is sufficiently large, the measurement function F generally defines a one-to-one correspondence between the attractor states in the full state space and the m-dimensional vector y. By "one-to-one" we mean that for a given y there is a unique x on the attractor such that y==F(x). When there is a one-to-one correspondence, each vector from m measurements is a 28

Chapter 2

Reconstruction of Signals

proxy for a single state of the system, and the fact that the information of the entire system is determined by a state x and F(x). In order for this to be valid, it suffices to set m to a value larger than twice the box-counting dimension of the attractor. The one-to-one property is useful because the state of a deterministic dynamical system is unique, and therefore its future evolution is completely determined by a point in the full state space. There are two types of embedding on the basis of measurements, which are relevant to system identification, namely, the topological embedding and differentiable embedding.

2.1.1

Topological Embeddings

Consider an n-dimensional Euclidean space

]Rn.

Points in this space are

defined by n real coordinate values, say Xl, X2,··· , Xn . Let us represent these elements by a vector Xoo [Xl X2··· xn]T. Let F be a continuous function from ]Rn

to

]Rm ,

where

]Rm

is an m-dimensional Euclidean space. The mapping

can be represented in the following way:

yooF(x),

(2.2)

whereYElRm. Let AclRn be an attractor ofa dynamical system. F(A) is an image of the attractor A in

]Rn via

the observation (measurement) function F.

F is bijective (one-to-one) if for any Xl, X2 C A, F(X1) = F(X2) implies Xl ooX2. For a bijective function F, there exists an inverse function F

-1.

A bijective

map on A which is continuous and has a continuous inverse is called topological embedding. In a typical experimental situation, the set A we are interested in is an attractor, which is a compact subset of

]R n

and is invariant

under the dynamical system. The goal is to use the measurements to construct the function F so that F(A) is a copy of A that can be analyzed. A finite time-series of measurements will produce a finite set of points that make up F(A). If enough points are present, we may hope to discern some of the properties of F(A) and therefore those of A. For topological embeddings we have the following theorem:

29

2.1

Reconstruction of System Dynamics

Theorem 2.1 [119]

A is a compact subset of JR" of box-counting dimension do. If m > 2do, then almost every d function F = [Ii h ... fm] T from JR nto JR mis a Assume that

topological embedding of A into JRm . The intuitive reason for the condition m > 2do can be seen by considering generic intersections of smooth surfaces in m-dimensional Euclidean space JRm . Two sets of dimensions d] and d2 in JRm mayor may not intersect. If they do intersect and the intersection is generic, then they will meet in a surface of dimension (2.3) If this number is negative, generic intersections do not occur. If the surface lies in a special position relative to one another, the intersection may be special and have a different dimension. In particular the delay coordinates can be used for constructing the topological embedding.

2.1.2

Delay Coordinates

Assume that our ability to make measurements of independent components is limited. In the worst case, we may be able to make measurements of a single scalar variable, say yet). Since the measurement depends only on the system state, we can represent such a situation by yet) = f(x(t)), where f is the single measurement function, evaluated when the system is in state xU). We assign to x(t) the delay coordinate vector H(xU)) = [yet - r)··· yet - mr)f =

[f(x(t - r))··· f(x(t - mr))f.

(2.4)

Note that for an invertible dynamical system, given x(t), we can view the state x (t - r) at any previous time t - r as being a function of x(t). This is true because we can start at x(t) and use the dynamical system to follow the 30

Chapter 2

Reconstruction of Signals

trajectory backward in time to the instant t - r. Hence, xCt) uniquely determines x (t - r), and thereby m is called embedding dimension. To emphasize this, we define hI (x(t»

= f(x(

t - r» , ... , hm(x(t» =f(x( t - m r» . Then, if

we writey(t) for [Yet - r) ···y(t- mr)f, we can express Eq. (2.1) as y=H(x),

(2.5)

where H(x) = [h](x) ... hm(X)]T is the delay coordinate function. The function H can be viewed as a special choice of the measurement function in Eq. (2.1). For this special choice, the requirement that the measurements do not lie in a special position is brought into question, since the components of the delay coordinate vector in Eq. (2.4) are constrained: They are simply time-delayed versions of the same measurement functionf This is relevant when considering small perturbations of H. Small perturbations in the measuring process are introduced by the scalar measurement function f, and they influence coordinates of the delay coordinate function H in an interdependent way. Although it was determined in the simultaneous measurement case that almost every perturbation of hI,···, hm leads to a one-to-one correspondence, these independent perturbations may not be achievable by perturbing the single measurement function

f

A simple example will illustrate this point. Suppose the set A contains a single periodic orbit whose period is equal to the delay time r . This turns out to be a bad choice of r, since each delay coordinate vector from the periodic orbit will have the form [h(x)··· h(x)]T for some x and lie along the straight line y] = ... =Ym in ]Rm . However, a circle cannot be continuously mapped to a line without points overlapping, violating the one-to-one property. Notice that this problem will affect all measurement functions h, so that perturbing h will not help. In this case, we cannot eliminate the problem by making the measurement function more generic; the problem is built-in. Although this case is a particularly bad one because of a poor choice of the time delay r , it shows that the reasoning for the simultaneous measurement case does not extend to delay coordinates, since it gives an obviously wrong conclusion in this case. 31

2.1

Reconstruction of System Dynamics

This problem can be avoided, for example, by perturbing the time delay 7 (if indeed that is possible in the experimental settings). In any case, some extra analysis beyond the geometric arguments we made for the simultaneous measurements case needs to be done. This analysis was tackled by Takens [1] and extended later by Sauer et al. [120]. The result can be stated as follows: Theorem 2.2

(Takens' embedding theorem with extensions by Sauer et al. [120])

Assume that a continuous-time dynamical system has a compact invariant set A (e.g., a chaotic attractor) of box-counting dimension do, and let m > 2do. Let

7

be the time delay. Assume that A contains only a finite number of

equilibria and a finite number of periodic orbits of period P T for 3 ｾ＠ P ｾ＠ m and that there are no periodic orbits of period 7 and 27. Then, with probability one, a choice of the measurement function h yields a delaycoordinate function H which is bijective from A to H(A). The one-to-one property is guaranteed to fail not only when the sampling rate is equal to the frequency of a periodic orbit, as discussed above, but also when the sampling rate is twice the frequency of a periodic orbit, that is, when A contains a periodic orbit of minimum period 27. To see why this is so, define the function ((x) = hex) - h( ¢-r (x)) on the periodic orbit, where ¢r denotes the action of the dynamics over time t. The function (is either identically zero or is nonzero for some x on the periodic orbit, in which case it has the opposite sign at the image point ¢-r (x) and changes sign on the periodic orbit. In any case, ((x) has a root Xo. Since the period

»

»

is 27, we have h( x o) = h(¢-r (x o = h( ¢-2r (x o = .. '. Then the delay coordinate map Xo and ¢-r (xo) are distinct, and H is not one-to-one for any observation function h. This problem may be eliminated by a proper choice of 7. Unfortunately, the choice of

7

that is not ruled out by the theory includes

those that are unnaturally small or large in comparison to the time constant of the system. Such values of

7

will cause the correlation between

successive measurements to be excessively large or small, causing the effectiveness of the reconstruction to degrade in real-world cases, where 32

Chapter 2

Reconstruction of Signals

noise is present. The choice of optimal time delay for unfolding the reconstructed attractor is an important and largely unresolved problem. A commonly used rule of thumb is to set the delay to be the time lag required for the autocorrelation function to become negative (zero crossing) or, alternatively, the time lag required for the autocorrelation function to decrease by a factor of expo Another approach, that of Fraser and Swinney [121], incorporates the concept of mutual information, borrowed from Shannon's information theory [122]. This approach provides a measure of the general independence of two variables, and chooses a time delay that produces the first local minimum of the mutual information of the observed quantity and its delayed value.

2.2

Differentiable Embeddings

Assume that A is a compact smooth d-dimensional submanifold of IRk. A circle is an example of a smooth one-dimensional manifold; a sphere and torus are examples of two-dimensional tangent space at each point. If F is a smooth function from one manifold to another, then the Jacobian matrix DF maps tangent vectors to tangent vectors. More precisely, for each point x on

A, the map DF(x) is a linear map from the tangent space at x to the tangent space at F(x). If, for all x in A, no nonzero tangent vectors map to zero under DF(x), then F is called an immersion. A smooth function F on a smooth manifold is called a differentiable embedding if F and F- 1 are one-to-one immersions. In particular, a differentiable embedding is automatically a topological embedding. In addition, the tangent spaces of A and F(A) are isomorphic. In particular, the image F(A) is also a smooth manifold of the same dimension as A. In 1936, Whitney [123] proved that if A is a smooth d-manifold in IRk and m> 2d, then a typical map from IRk to IR m is a differentiable embedding when restricted to A. Takens [1] proved a result in this context for delay coordinate functions. The following is a version of Takens' theorem by Sauer et al. [120]. 33

2.3

Phase Space Reconstruction - Example

Theorem 2.3 (Sauer et al. [120]) Assume that a continuous time dynamical system has a compact invariant smooth manifold A of dimension d, and let m > 2d. Let , be the time delay. Assume that A contains only a finite number of equilibria, a finite number of periodic orbits of period p, for 3 :( p :( m, and that there are no periodic orbits of period, and 2, . Assume that the Jacobians of the return maps of those periodic orbits have distinct eigenvalues. Then, with probability one, a choice of the measurement function h yields a delay coordinate function H which is bijective from A to H(A).

From the point of view of extracting information from an observed dynamical system, there are some advantages of having a differentiable embedding, as compared to a topological embedding. These advantages stem from the fact that metric properties are preserved by the reconstruction. Differentiable embeddings offer two advantages compared to topological ones [119]. First, there is a uniform upper bound on the stretching done by H 1 and H- • Such H functions are referred to as bi-Lipschitz. The dimensions are preserved under bi-Lipschitz maps. Second, all Lyapunov exponents on an attractor are reproduced in the reconstruction. For the purpose of the present study, we are interested in the consequences of the embedding theorems for the synchronization/transmission problem. These consequences can be summarized in the following result. (Main existence theorem [124]) If the assumptions of the embedding theorems are satisfied, it is always possible to reconstruct the state of the system and synchronize (e.g., by forcing the states) an exact copy of it on the basis of the measurements of a single (scalar) output variable.

Theorem 2.4

2.3

Phase Space Reconstruction-Example

From a signal processing perspective, an issue of paramount importance is the reconstruction of dynamics from measurements made on a single coordinate of the system. The motivation here is to make "physical sense" from the 34

Chapter 2

Reconstrnction of Signals

resulting time-series, by passing a detailed mathematical knowledge of the underlying dynamics. where N is the total number Let the time-series be denoted by ｻｳＨｮｔＩｽｾｯＧ＠ of samples and T is the sampling period. To reconstruct the dynamics of the original attractor that gives rise to the observed time-series, we seek an embedding space where we may reconstruct an attractor from the scalar data so as to preserve the invariant characteristics of the original unknown attractor [125, 126]. For simplicity we set T= 1. By applying the delay coordinate method, the dEx 1 phase space vector sen) is constructed by assigning coordinates: SI

(n) = sen),

S2 (n) =

sen - r),

(2.6)

where dE is the embedding dimension and r is the normalized embedding delay. These parameters are not chosen arbitrarily; rather they have to be determined experimentally. Procedures for estimating the normalized embedding delay r based on mutual information are described in the literature [121 - 129]. To estimate the embedding dimension dE,we may use the method of global false nearest neighbor [130 - 133]. Another important parameter that needs to be determined in the analysis of an observed process is the local or dynamical dimension [130]. The local dimension represents the number of dynamical degrees of freedom that are active in determining the evolution of the system as it moves around the attractor [130, 131]. This integer dimension gives the number of true Lyapunov exponents of the system under investigation. It is less than or equal to the global embedding dimension; that is, dL ｾ､ｅＧ＠ The parameter dLcan be determined by using the local false-nearest-neighbor (LFNN) method [130, 131]. The notion offalse nearest neighbors refers to nearest neighbors as attractors that have become near one another through projection when the attractors are viewed in a dimension too low to unfold the attractors completely. Let us analyse an example of reconstruction of an attractor from a 35

2.3

Phase Space Reconstruction - Example

measured time-series obtained from the numerical integration of Chua's circuit (see Fig. 2.l) [134 - 138]. The circuit will also be used in subsequent chapters of this book. R

L

(a)

v

(b)

Figure 2.1 Chua's Circuit. (a) Circuit topology and (b) Chua's diode characteristic

The circuit is certainly one of the most widely studied nonlinear circuits and a great number of papers ensure that the dynamics of this circuit are also well documented. The normalized equations of the circuit are

X2 = XI - x 2 + x3' X3 = a 2 x2 , where

K(XI )

(2.7)

is a piecewise-linear function given by mIXI K(XI ) =

+ (rna

- ml )

for

XI ｾ＠

maxI

for 1XI

mlxl - (rna - m l )

for

{

1

1< 1

(2.8)

xI::::;-1

with mo= -1/7 and ml = 2/7. For different values of a l and a 2 , the system operates in different regimes, e.g., periodic and chaotic regimes. The wellknown double scroll Chua's attractor, for example, is obtained fora l = 9 and a2 = -100/7. The largest Lyapunov exponent and the Lyapunov dimension 36

Chapter 2

Reconstruction of Signals

of the attractor are equal to 0.23 and 2.13, respectively [139,140]. Equations (2.7) and (2.8) are simulated using a fourth-order Runge-Kutta algorithm with integration step size equal to 10-2 • Figures 2.2 (a) and 2.2(b) 0.4 0.3 0.2 0.1 0 ｾ＠

-0.1 -0.2 -0.3 -0.4 -2.5

-2

-1.5

-I

-0.5

2.5

0 Xl

(a)

4 3 2

N

0 -1

-2 -3

-4 5

0.5 0 X

0 -5 -0.5

Y

(b)

Figure 2.2 Projection of the double scroll attractor observed in Chua's circuit on (a) x - y (i.e., Xl -

X2)

plane, and (b) X - Y - z (i.e., Xl -

X2 - X3)

space 37

2.3

Phase Space Reconstruction - Example

show the two-dimensional projection of the double scroll attractor on the x - y (or x) - X2) plane and the three-dimensional attractor in ]R3 . To be able to reconstruct the dynamics we further use only recording of i L , i.e., X3 (normalized current). To be able to apply the delay coordinate method we need to find a suitable time delay. In this example we follow the mutual information criteria to choose the delay T . It is found that T = 17 [124]. Figure 2.3 shows the reconstructed attractors on the x - y plane for different delays by using dE = 6. When the delay is too small, the reconstructed attractor becomes compacted and the dynamics is not well visualized. Figure 2.3(a) gives an example with the delay chosen as 4,------------------------,

T

= 3. Figure 2.3(b) shows the

4

2

-,

0

"-,

0

-)

-)

-2

-2

-3

-3

-4

-2

0 x

2

-4

4

-2

(a)

2

4

(b)

4

4

"-,

"

-)

-)

-2

-2

-3

-3

-4

0 x

-2

0 x

(c)

2

4

-4

-2

4 x

(d)

Figure 2.3 Reconstructed double scroll attractors using measured iL or X3 (i.e., z) time-series. Successive figures show attractors reconstructed using time delays of 3, 17, 30, 50, respectively. When the time delay is too small, the reconstructed attractor is squeezed. When it is too large, the reconstruction is bad - the geometric structure of the attractor is lost. With a proper choice of the delay time, as seen in (b), the double scroll structure is well reproduced

38

Chapter 2

reconstructions with

T

=

Reconstruction of Signals

17 as detennined from the mutual infonnation

criteria. In this case the attractor structure is easily seen. When the delay used for reconstruction becomes too large, the delayed copies of the signal become more and more uncorrelated. This situation is indicated in Figs. 2.3 (c) and (d). The trajectories shown in these two figures do not resemble at all the original double scroll structure. Figure 2.4 shows the reconstructed attractor in the reconstruction space by using attractor structure can be seen clearly.

T

= 17 and dE = 6. In this case, the

4 3 2

:;. 0 -1

-2 -3

-4 -5

o Figure 2.4 Reconstructed Chua's double scroll attractor in the reconstruction space by using T = 17 and dE = 6

2.4

Problems and Research Approaches

Embedding theorems offer only the existence result. Construction of an inverse of an embedding function is an open problem-no general solution or algorithm is available [141]. In principle, any choice of the time delay

T

is acceptable in the limitation

of an infinite amount of noiseless data. In the more likely situation of a finite amount of data, the choice of

T

is of considerable practical importance in

trying to reconstruct the attractor that represents the dynamical system 39

2.4

Problems and Research Approaches

which has generated the data. As mentioned above, if the time delay r is too short, the coordinates sen) and sen + r) that we wish to use in our reconstructed data vector sen) will not be independent enough. That is to say, not enough time will have evolved for the system to explore enough of its state space to produce, in a practical numerical sense, new information about that state space. On the other hand, since chaotic systems are intrinsically unstable, if r is too large, any connection between the measurements sen) and sen + r) is numerically equivalent to being random with respect to each other, which makes the reconstruction of the system dynamics very difficult. Even though it has not been stated clearly in the embedding theory, the reconstructions in the context of the Takens' embeddings are valid for autonomous systems only. In addition, the results stated in the previous sections are closely related to the observability problem known from control theory. Observability issues are well developed for linear systems, but only a limited number of results for nonlinear cases exist. Observers provide "the missing tool" for reconstruction. In Chapter 4, we will expand the results here to include non-autonomous systems based on the observer approach. Measurement noise, which always companies the observed chaotic data, contaminates the original chaotic signal and destroys the basic dynamics of the system of interest. Identifying and separating chaotic signal from noisy contaminated signal is still a fundamental issue in the context of chaotic signal processing and chaos-based communications [151]. Chapter 5 - 8 will discuss the modeling of chaotic systems via noisy measured data based on neural network approaches.

40

Chapter 3

3.1

Fundamentals of Neural Networks

Motivation

Artificial neural networks, commonly referred to as "neural networks", have been motivated right from their inception by the recognition that the human brain computes in an entirely different way from the conventional digital computer. The brain is a highly complex, nonlinear and parallel computer (information-processing system). It has the capability to organize its structural constituents, known as neurons, so as to perform certain computations (e.g., pattern recognition, perception, motor control, etc) many times faster than the fastest digital computer in existence today. Consider, for example, human vision, which is an information-processing task [175]. It is the function of the visual system to provide a representation of the environment around us and more importantly, to supply the information we need to interact with the environment. To be specific, the brain routinely accomplishes perceptual recognition tasks (e.g., recognizing a familiar face embedded in an unfamiliar scene) in approximately lOO - 200 ms, whereas tasks of much less complexity may take days on a conventional computer. How, then, does a human brain do it? At birth, a brain has great structure and the ability to build up its own rules through what we usually refer to as "experience". Indeed, experience is built up over time, with the most dramatic development (i.e., hard-wiring) of the human brain taking place during the first two years from birth, but the development continues well beyond that stage. A "developing" neuron is synonymous with a plastic brain: plasticity permits the developing nervous system to adapt to its surrounding

3.1

Motivation

environment. Just as plasticity appears to be essential to the functioning of neurons as information-processing units in the human brain, so it is with neural networks made up of artificial neurons. In its most general form, a neural network is a machine that is designed to model the way in which the brain performs a particular task or function of interest; the network is usually implemented by using electronic components or is simulated in software on a digital computer. In this book we confine ourselves to an important class of neural networks that perform useful computations through a process of learning (adaption). To achieve good performance, neural networks employ a massive interconnection of simple computing cells referred to as "neurons" or "processing units". We may thus offer the following definition of a neural network viewed as an adaptive machine: Definition 3.1 [176] A neural network is a massively parallel distributed processor made up of simple processing units, which has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: (1) Knowledge is acquired by the network from its environment through a learning process. (2) Interneuron connection strengths, known as synaptic weights, are used to store the acquired knowledge. The procedure used to perform the learning process is called a learning algorithm, the function of which is to modify the synaptic weights of the network in an orderly fashion to attain a desired design objective. The modification of synaptic weights provides the traditional method for the design of neural networks. Such an approach is the closest to linear adaptive filter theory, which is already well established and successfully applied in many diverse fields [177, 178], However, it is also possible for a neural network to modify its own topology, which is motivated by the fact that neurons in the human brain can die and that new synaptic connections can grow.

42

Chapter 3

3.2

Fundamentals of Neural Networks

Benefits of Neural Networks

It is apparent that a neural network derives its computing power through,

first, its massively parallel distributed structure, and second, its ability to learn and therefore generalize. Generalization refers to the neural network producing reasonable outputs for inputs not encountered during training (learning). These two information-processing capabilities make it possible for neural networks to solve complex (large-scale) problems that are currently intractable. In practice, however, neural networks cannot provide the solution by working individually. Rather, they need to be integrated into a consistent system engineering approach. Specifically, a complex problem of interest is decomposed into a number of relatively simple tasks, and neural networks are assigned a subset of the tasks that match their inherent capabilities. The use of neural networks offers the following useful properties and capabilities [179]: (1) Nonlinearity

An artificial neuron can be linear or nonlinear. A

neural network, made up of an interconnection of nonlinear neurons, is itself nonlinear. Moreover, the nonlinearity is special in that it is distributed throughout the network. Nonlinearity is a crucial property, particularly if the underlying physical mechanism being responsible for generation of the input signal (e.g., speech signal) is inherently nonlinear. (2) Input-Output

Mapping A popular paradigm of learning called

learning with a teacher or supervised learning involves modification of the synaptic weights of a neural network by applying a set of labeled training samples or task examples. Each example consists of a unique input signal and a corresponding desired response. The network is presented with an example picked at random from the set, and the synaptic weights (free parameters) of the network are modified to minimize the difference between the desired response and the actual response of the network produced by the input signal in accordance with an appropriate statistical criterion. The training of the network is repeated for many examples in the set until the 43

3.2

Benefits of Nenral Networks

network reaches a steady state where there are no further significant changes in the synaptic weights. The previously applied training examples may be reapplied during the training session but in a different order. Thus, the network learns from the examples by constructing an input-output mapping for the problem at hand. Such an approach brings to mind the study of nonparametric statistical inference, which is a branch of statistics dealing with model-free estimation. The term "nonparametric" is used here to signify the fact that no prior assumptions are made on a statistical model for the input data. (3) Adaptivity

Neural networks have a built-in capability to adapt their

synaptic weights to changes in the surrounding environment. In particular, a neural network trained to operate in a specific environment can be easily retrained to deal with minor changes in operating environmental conditions. Moreover, when it is operating in a nonstationary environment (i.e., one where statistics change with time), a neural network can be designed to change its synaptic weights in real time. The natural architecture of a neural network for pattern classification, signal processing, and control applications, coupled with the adaptive capability ofthe network, makes it a useful tool in adaptive pattern classification, adaptive signal processing, and adaptive control. As a general rule, it may be said that the more adaptive we make a system, all the time ensuring that the system remains stable, the more robust its performance will likely be when the system is required to operate in a non stationary environment. It should be emphasized, however, that adaptivity does not always lead to robustness; indeed, the opposite may occur. For example, an adaptive system with short time constants may change rapidly and therefore tend to respond to spurious disturbances, causing a drastic degradation in system performance. To realize the full benefits of adaptivity, the principal time constants of the system should be long enough for the system to ignore spurious disturbances and yet short enough to respond to meaningful changes in the environment. The problem described here is referred to as the stability-plasticity dilemma [180].

44

Chapter 3

(4) Evidential Response

Fundamentals of Neural Networks

In the context of pattern classification, a neural

network can be designed to provide infonnation not only about which particular pattern to select, but also about the confidence in the decision made. This latter infonnation may be used to reject ambiguous patterns, should they arise, and thereby improve the classification perfonnance of the network. (5) Contextual Information

Knowledge is represented by the very

structure and activation state of a neural network. Every neuron in the network is potentially affected by the global activity of all other neurons in the network. Consequently, contextual infonnation is dealt with naturally by a neural network. (6) Fault Tolerance

A neural network, implemented in hardware fonn,

has the potential to be inherently fault tolerant, or capable of robust computation, in the sense that its perfonnance degrades gracefully under adverse operating conditions. For example, if a neuron or its connecting links are damaged, recall of a stored pattern is impaired in quality. However, due to the distributed nature of infonnation stored in the network, the damage has to be extensive before the overall response of the network is seriously degraded. Thus, in principle, a neural network exhibits a graceful degradation in perfonnance rather than catastrophic failure. There is some empirical evidence for robust computation, but usually it is uncontrolled. In order to be assured that the neural network is in fact fault tolerant, it may be necessary to take corrective measures in designing the algorithm used to train the network. (7) VLSI Implementability

The massively parallel nature of a neural

network makes it potentially fast for the computation of certain tasks. This same feature makes a neural network well suited for implementation using very-Iarge-scale-integration (VLSI) technology. One particular beneficial virtue of VLSI is that it provides a means of capturing truly complex behavior in a highly hierarchical fashion. (8) Uniformity of Analysis and Design

Basically, neural networks enjoy

universality as infonnation processors. We say this in the sense that the 45

3.3

Radial Basis Function Neural Networks

same notation is used in all domains involving the application of neural networks. This feature manifests itself in different ways:

CD

Neurons, in one form or another, represent an ingredient common to

all neural networks.

®

This commonality makes it possible to share theories and learning

algorithms in different applications of neural networks.

®

Modular networks can be built through a seamless integration of

modules. (9) Neurobiological Analogy

The design of a neural network is moti-

vated by analogy with the brain, which is a living proof that fault tolerant parallel processing is not only physically possible but also fast and powerful. Neurobiologists look to (artificial) neural networks as a research tool for the interpretation of neurobiological phenomena. On the other hand, engineers look to neurobiology for new ideas to solve problems more complex than those based on conventional hard-wired design techniques. The field of neural networks is now extremely vast and interdisciplinary, drawing interest from researchers in many different areas such as engineering (including biomedical engineering), physics, neurology, psychology, medicine, mathematics, computer science, chemistry, economics, etc. Artificial neural networks provide a neurocomputing approach for solving complex problems that might otherwise not have tractable solutions. In the following, we will briefly review the fundamentals of the two classes of neural networks, i.e., radial basis function (REF) neural networks, and recurrent neural networks (RNNs).

3.3 3.3.1

Radial Basis Function Neural Networks Background Theory

Originally, the REF method has been traditionally used for data interpolation in multi-dimensional space [181- 183]. The interpolation problem can be 46

Chapter 3

Fundamentals of Neural Networks

stated as follows: Given a set of N different points {z;

E

IR M Ii = 1, ... , P} and a corres-

IRI Ii = 1,,,,, P}, find a function h: IR M ---+ IRI that satisfies the interpolation condition:

ponding set of P real numbers {d;

E

(3.1)

h(zJ=di'i=l, .. ·,P.

Note that for strict interpolation as specified here, the interpolation surface (i.e., function h) is constrained to pass through all the training data points. The RBF technique consists of choosing a function h that has the following form: p

h(z) =

L w;qJ(llz - z;ll),

(3.2)

i=l

where {qJ(llz-z;II)Ii=l, ... ,P} is a set of P arbitrary (generally nonlinear) functions, known as RBFs, and 11'11 denotes here a norm that is usually taken to be Euclidean [152]. The known data points Z; EIR M ,i=1,2, .. ·,P are taken to be the centers of the RBF (see Fig. 3.1). Inserting the interpolation conditions of Eq.(3.1) in Eq.(3.2), we obtain the following set of simultaneous linear equations for the unknown coefficients (weights) of the expansion {Wi}:

qJIP] [WI qJ2P w2

·· ·

.. .

qJpp

Wp

_

-

[dl] d2 . , ..

(3.3)

dp

where

qJij=qJ(llz;-zjID, j,i=1,2, .. ·,P.

(3.4)

Let

d=[dl d 2

w=

[WI

...

w2

...

dpf, Wp]T.

(3.5) 47

3.3

Radial Basis Function Neural Networks

The P-dim vector d and w represent the desired response vector and linear weight vector, respectively. Let (/J denote an P x P matrix with elements rpji: (3.6)

(/J = {rpji I j, i = 1,2, "', P}.

We call this matrix the interpolation matrix. We may then rewrite Eq. (3.3) in the compact form

(/Jw

=

d.

(3.7)

Assuming that (/J is nonsingular and therefore that the inverse matrix (/J-l exists, we may go on to solve Eq. (3.7) for the weight vector w, obtaining (3.8) The vital question is: How can we be sure that the interpolation matrix (/J is nonsingular? It turns out that for a large number of radial-basis functions and under certain conditions, the answer to this question is given in the following important theorem. be distinct inIRM. Then the P-by-P

Theorem 3.1 [181] Let

Z\'Z2, "',Zp

interpolation matrix

whose ji-th element is CfJJi

(Jj

=

rp (liz) - zilD is

nonsingular. There are a class of radial basis functions that are covered by the above theorem, which includes the following functions that are of particular interest in the study of radial-basis function networks:

(1) Thin plate spline function 2

x log (x).

(3.9)

(2) Multiquadrics [184]

rp(r) = (r2 + C2)1/2

for some c> 0,

and

rE JR.

(3.10)

for some c> 0,

and

rER

(3.11 )

(3) Inverse multiquadrics [184]

rp(r)

48

=

(2 2)1/2 r +c

Chapter 3

Fuudamentals of Neural Networks

(4) Gaussian functions rp(r) = exp ( -

;;2 J

for some

(J>

0,

and

rE

R

(3.12)

where r represents the Euclidean distance between a center and a data point. The parameter

(J,

termed spread parameter, controls the radius of influence

of each basis function. The multi-quadratic and the Gaussian kernel functions share a common property-they are both localized functions, in the sense that rp --+

°

as r --+

00.

For the radial basis functions listed in

Eq. (3.9) to Eq. (3.12) to be nonsingular, the points {ZJ:l must all be different (i.e., distinct). This is all that is required for non singularity of the interpolation matrix (/J, regardless of the number N of the data points or dimensionality M of the vectors (points) Zi.

3.3.2

Research Progress in Radial Basis Function Networks

Broomhead and Lowe [185] were the first to exploit the use of the radial basis function (RBF) in the design of neural networks. They pointed out that the procedure of the RBF technique for strict interpolation may not be a good strategy for the training of RBF networks for certain classes of tasks because of poor generalization to new data. Specifically, when the number of data points in the training set is much larger than the number of degrees of freedom of the underlying process, and we are constrained to have as many RBFs as the data points presented, the problem is overdetermined. Consequently, the network may end up fitting misleading variations due to noise in the input data, thereby resulting in a degraded generalization performance. Broomhead and Lowe [185] removed the restriction for strict function interpolation and set up a two-layer network structure where the RBFs are employed as computation units in the hidden layer. They considered that the training phase of the network learning process constitutes the optimization of a fitting procedure for a desired surface, based on the known data points presented to the network in the form of input-output examples 49

3.3

Radial Basis Function Neural Networks

(patterns). The generalization phase of the network learning process was considered synonymous with interpolation between the data points. The interpolation being performed along the constrained surface generated by the fitting procedure was viewed as an optimal approximation to the true surface. Shown in Fig. 3.1 is the prototype of the basic RBF neural network, in which the computational units (referred to here as hidden neurons or hidden units) provide a set of "functions" that constitutes a "basis" for the network input vectors when they are expanded into the hidden unit space. The output of the RBF network is a linear combination of the outputs from its hidden units. z 21

CPi ""

cp(lz-zil)

CPI

WI Z2

I:

11'}

h(z)

zM-1

Output layer WN

zM

Input layer

Hidden layer

Figure 3.1 The architecture of an RBF network. The hidden unit

CPi

is centered at Zi'

The dimensionality of the input space is M and that of the hidden unit space is N

Poggio and Girosi [186] treated the problem of neural networks for approximation in a theoretical framework, based on regularization techniques. They highlighted the application of the traditional regularization techniques and derived the network structures which they called regularization networks. They considered an RBF neural network as a special case in the realm of regularization networks. By this approach, it is also possible to reveal what happens inside RBF networks. The network of Poggio and Girosi has two layers and the number of hidden units is fixed apriori. The centers of the hidden units are a subset of the input samples. The number of the network 50

Chapter 3

Fundamentals of Neural Networks

input is equal to the number of independent variables of the problem. The approximation of a continuous function depends on finding the linear weights connecting the hidden layer and output layer. The distance between the function constructed by the neural network and the desired function is measured by a cost function. The gradient descent algorithm can be used to find suitable values for the weights in order to obtain good approximation of the underlying function. Before applying the gradient algorithm, a suitable initialization is given for the output weights. In order to decrease the computational complexity required for finding exact solutions when the sample size is large, an approximation to the regularized solution was introduced by Poggio and Girosi [186]. Based on the "locally-tuned" concept of neurons in various biological nervous systems, Moody and Darken [187] presented a network containing computational units with locally-tuned response functions. This network model takes the advantages of local methods, which have been originally used for density estimation classification, interpolation and approximation. According to Moody and Darken [187], the local methods have attractive computational properties including parallelizability and rapid convergence. Due to the locality of the unit response, for any given input, only a small fraction of computational units with centers very close to the input (in the input space) will respond with activation which differs significantly from zero. Thus, only those neurons with centers close enough to the input need to be evaluated and trained. The structure of the proposed network is the same as an RBF network. A radially symmetric function, which has a single maximum at the origin and drops rapidly to zero at large radii is chosen as the RBF in the sole hidden layer of the network. For this network, the number of hidden units is chosen apriori. The RBF hidden neuron centers are chosen as a random subset of the available training patterns and no special assumptions are made about this selection such as orthogonality or being uniformly distributed over the input space. Two types of training algorithms were studied by Moody and Darken [187]. One is a fully supervised method and the other is a hybrid method combining the 51

3.3

Radial Basis Function Neural Networks

supervised and self-organizing methods. Comparison of the performance of the fully supervised method and the back propagation network shows that the supervised method does not learn appreciably faster than the back propagation network because it casts the learning as a nonlinear optimization problem which results in slow convergence. Furthermore, since this method assumes large width values for the Gaussian functions, it loses the locality desired for the computational units in the network. In the hybrid method, a self-organized learning scheme, which is the standard k-means clustering algorithm, is used to estimate the centers of the basis functions, and the width values are computed using various "P nearest-neighbor" heuristics. A linear supervised learning method (the standard least mean square (LMS) algorithm) is employed to estimate the output weights. Because the local response of the network is ensured by this hybrid algorithm, only a few hidden units respond to any given input, thus reducing the computational overhead. In addition, adaptation of the network parameters in this algorithm is linear rather than nonlinear, leading to fast convergence. Clearly, this hybrid learning algorithm is faster than the back propagation method. As pointed out by Poggio and Girosi [186], there are two basic problems associated with the k-means clustering based algorithm. The first problem is that there is still an element of chance in getting the right hidden neuron centers. Second, since clustering is more probability density oriented, this method is suitable for the problem of pattern classification, whereas for function approximation, clustering may not guarantee good results, because two samples close to each other in the input space do not necessarily have similar outputs. To overcome the problems, Chen et af. [188] proposed a more systematic procedure, known as orthogonal forward regression (OFR) to select the RBF hidden neuron centers. This iterative method was derived by augmenting and modifying the well-known orthogonal least squares (OLS) algorithm. In this procedure, a Gram-Schmidt type of orthogonal projection is employed to select the best center one at a time. The selection process terminates when 52

Chapter 3

Fundamentals of Neural Networks

the number of predetermined centers has been filled or the benefit of adding more hidden neurons becomes diminishingly small. So far, the learning algorithms we have described are based on the assumption that the number of hidden units is chosen apriori. During the application of the above algorithms, users are often puzzled by the problem of choosing the right number of hidden units. It may not be possible to train an RBF neural network to reach a desired level of performance if the network does not have enough hidden neurons, or the learning algorithm fails to fmd the optimal network parameters. Therefore, it is useful to develop a new type of learning algorithm capable of automatically recruiting new hidden units whenever necessary for improving the network performance. A hierarchically self-organizing learning (HSOL) algorithm for RBF was developed in [189]. This algorithm was capable of automatically recruiting new computational units whenever necessary for improving the network performance. In the HSOL algorithm, the hidden units are associated with the accommodation boundaries defined in the input space and the class representation defined in the output space. The accommodation boundary of a hidden unit defines a region of input space upon which the corresponding hidden unit has an influence. If a new sample falls within the accommodation boundary of one of the currently existing hidden units, which has the same class representation as that of the new sample, then the network will not generate a new hidden unit but accommodate the new sample by updating the parameters of the existing hidden units. Otherwise, the network will recruit a new hidden unit. Furthermore, in HSOL, the accommodation boundaries of individual hidden units are not fixed but adjusted dynamically in such a way as to achieve hierarchical learning. Initially, the accommodation boundaries are set large for achieving rough but global learning, and gradually reduced to a smaller size for fine learning. In summary, the HSOL starts with learning global mapping features based on a small number of computational units with larger accommodation boundaries and then proceeds to learn finer mapping details with increasing number of computational units and diminishing accommodation boundaries. The HSOL algorithm starts 53

3.3

Radial Basis Function Neural Networks

from no hidden units and then builds up the number of hidden units from the input data. An alternative approach is to start with as many hidden units as the number of inputs and then reduce them using a clustering algorithm which essentially puts close patterns in the input space into a cluster to remove the unnecessary hidden units. Such an approach has been developed by Musavi et al. [190] and it essentially consists of an iterative clustering algorithm that takes class membership into consideration and provides an approach to remove the unnecessary hidden units. This approach also provides for the width estimation of the basis functions. After the training patterns have been clustered and their centers are found, the algorithm selects the width parameters or the Gaussian functions in the hidden layer. This is done by fmding the eigenvalues inside the variance matrix of each Gaussian function. The Gram-Schmidt orthogonalization procedure is utilized to determine the normalized axes of the contour or the constant potential surface of the corresponding Gaussian function and the projections of the nearest neighbor of the opposite class on these axes. The eigenvalues can then be derived from these projections. In practical applications of an RBF neural network, learning results from many presentations of a prescribed set of training examples to the network. One complete presentation of the entire training set during the network learning process is called an epoch. There are two types of learning. One is called batch learning. To perform batch learning, the network parameters are updated after the presentation of all training patterns which constitute an epoch. So the batch learning process is maintained on an epoch-by-epoch basis until the parameters of the network stabilize and the average network output error over the entire training set converges to some minimum value. All the learning algorithms mentioned above belong to batch learning. The other type oflearning is sequential (on-line/recursive) learning, in which the network parameters are adjusted after the presentation of each training pattern. To be specific, consider an epoch consisting of N training examples (patterns) arranged in the order [ZI' d l ],

... , [ZN'

dN ]. The first example [z], d l ] in the

epoch is presented to network and the network output is calculated. The 54

Chapter 3

Fundamentals of Neural Networks

network parameters are then updated. After this, the second example

[Z2,

d2 ]

is presented and the above procedure is repeated resulting in further adaptation of the network parameters. This process is continued until the last example

[ZN'

dN ] is accounted for. In the rest of the section, we look at some

sequential learning algorithms. In addition to OFR algorithm mentioned above, Chen et al. have proposed a recursive hybrid learning algorithm for RBF neural networks [191]. They applied such an RBF neural network for on-line identification of nonlinear dynamical systems. Their algorithm employs a hybrid clustering and least squares algorithm. The recursive clustering algorithm adjusts the hidden neuron centers for the RBF network while the recursive least squares algorithm estimates the connection weights. Thin-plate-spline functions are chosen as the RBFs in the network and only the hidden unit center values have to be determined in the network hidden layer. However, in this algorithm, the number of hidden units has to be determined before training commences, and this number varies from application to application. To remedy this problem, Platt proposed a sequential learning algorithm for a resource allocation network (RAN) [192]. This algorithm is inspired by the method of a harsh table lookup, which was proposed by Moody [187]. Based on the idea of adjusting the number of hidden units to reflect the complexity of the function to be interpolated, hidden neurons are added based on the "novelty" (referred to as "innovations" in the estimation literature) of the network input data. The network parameters are then estimated using the well-known least mean square (LMS) algorithm. A new pattern is considered novel if that input pattern is far away from the existing centers and if the error between the network output and the desired output is large. If no additional hidden neuron is added, the parameters of the existing hidden neurons, such as the centers, widths and weights, are updated by the LMS algorithm. Kadirkamanathan and Niranjan interpreted Platt's RAN from the viewpoint of function space [193]. However, it should be noted that even with the learning algorithm developed by Platt [193], it is still difficult to achieve a minimal RBF network. This is due to the following senous 55

3.4

Recurreut Neural Networks

drawback of REF networks. Once a hidden unit is created, it can never be removed. Because of this, REF networks could produce networks in which some hidden units, although active initially, may subsequently end up contributing little to the network output. Thus, pruning becomes imperative for the identification problems of nonlinear systems with changing dynamics, because failing to prune the network in such cases will result in the presence of numerous inactive hidden neurons. If inactive hidden units can be detected and removed as learning progresses, a more parsimonious network topology can be realized. In this book, we will consider a modified REF network with a learning algorithm that combines the pruning strategy to minimize the network. In order to ensure that transition in the number of hidden neurons is smooth, the novelty criteria in RAN are augmented with an additional growth criterion based on the root mean square value of the output error over a sliding data window. In brief, the new algorithm developed in this book improves the basic method of hidden neuron growth and network parameter adaptation by adding a pruning strategy to achieve a minimal REF network. Details will be given in Chapter 7.

3.4 3.4.1

Recurrent Neural Networks Introduction

Feedforward neural networks (FNNs) have been known to have powerful capability for processing static information [194], function approximation [176], etc. When temporal information is concerned, we can either add time delay steps to spatialize the temporal signal or use recurrent networks to process the temporal information. Recurrent neural networks (RNN s) are characterized by adding recurrent connections to the feedforward networks. The feedback links provide extra capability to process temporal information [195]. Under certain conditions, the RNN s can also make generalization from the training data to produce smooth and consistent dynamical behavior for 56

Chapter 3

Fundamentals of Neural Networks

entirely new inputs or new regions of the state space (i.e., inputs or regions of the state space not encountered during training). RNN network techniques have been applied to a wide variety of problems. Simple partially RNNs were introduced in the late 1980's by Rumelhart et al. [196] to learn strings of characters. Recently, many other applications have been reported involving dynamical systems with time sequences of events. For example, the dynamics of tracking the human head for virtual reality systems was studied by Saad [197]. The forecasting of financial data and of electric power demand were the objects of other studies [198, 199]. RNNs were also used to track water quality and to minimize the additives needed for filtering water [200].

3.4.2

Topology of the Recurrent Networks

On the research of the topology of the recurrent networks, much work has been done on the analysis and comparison of the location of the feedback connections. In general, the treatment on the feedback terms can be categorized into three approaches: the locally recurrent globally feedforward approach, the context unit approach and the global feedback approach. In the first approach, Back and Tsoi [201] suggested an architecture with the time delays and the feedback terms similar to those in infinitive impulse response (UR) and finitive impulse response (FIR) filters, all occurring inside a synapse. Frasconi et al. [202] allow self-feedback connections in the dynamical neurons only. There are no inter-neuron feedback terms in these models but multiple delays are allowed in each feedback. Tsoi and Back [203] named this approach local recurrent and global feedback architecture. Alternatively, inter-neural feedbacks are present in the other two approaches. Examples of the context unit approach are the Jordan network shown in Fig. 3.2 and the Elman network in Fig. 3.3. Jordan [204] copied the activation of the output units into the context units. The context units, together with self-feedback loops, link feedforwardly to the hidden units. These feedback links let the hidden layer interpret the previous state of the network and thus 57

3.4

Recurrent Neural Networks

provide the network with memory. Elman [205], on the other hand, copied the activation of the hidden units to form the context units. The hidden units appear to encode both the present input information and the previous states represented by the hidden units. The feedback connections to the context units are non-trainable and thus the generalized delta rule can be used for the training of the whole network. As errors are not backpropagating through time, long term dependence on time is difficult to model. Output

Output

Input units

Input units

Input

Input

Figure 3.2 A Jordan network

t

Figure 3.3 An Elman network

Finally, the global feedback approach allows inter-neuron feedback. William and Zipser [206] derived the real-time recurrent learning (RTRL) algorithm using a fully connected structure as shown in Fig. 3.4. All noninput units, including the output units and hidden units, are fully feedbacked by the recurrent weights. Robinson and Fallside [207] used a similar architecture but only the hidden units are fully connected with the recurrent weights.

3.4.3

Learning Algorithms

The learning algorithm defmes how the connection weights are being updated. The generalized delta rule for back propagation network is generally not applicable directly as the output activations are recursively dependent on the 58

Chapter 3

Fuudamentals of Neural Networks

Output

v

Figure 3.4 Architectural graph of a real-time recurrent network

recurrent connections. A simple modification is the back propagation through time (BPTT) by Rumelhart et al. [196]. Unfortunately this learning algorithm is not local in time and the amount of storage needed varies with the length of the training samples. The BPTT will be a good choice if the training sequences are known, in advance, to be short in length. Another well-known learning rule for the recurrent network is the RTRL by Williams and Zipser [206] which uses the sensitivity matrix to provide an on-line learning algorithm of the recurrent networks. There are some other learning rules, such as the one by Schmidhuber [208], which combines the BPTT and the RTRL to form a learning algorithm. Another variation is the subgrouped RTRL proposed by Zipser [209], which divides the fully recurrent network into sub-groups. The connections between different subnets are fixed and nontrainable. In Chapter 8, we will use a modified RNN to realize the equalization task in chaos-based communication systems.

59

Chapter 4

Signal Reconstruction in N oisefree and Distortionless Channels

In Chapter 2, it has been shown that embedding reconstructions in the context of the Takens' embedding theory are applicable only for timeinvariant systems. It is shown in this chapter that embedding reconstructions can be applied to time varying systems based on an observer approach. In particular, we consider the Lur' e system. As an application example, we discuss the information retrieval in chaos-based communication systems in the absence of noise and distortion.

4.1

Reconstruction of Attractor for Continuous TimeVarying Systems

Consider the chaotic system

x = g(x,t), where

X= [Xl Xz ... XD]T

(4.l)

is the D-dimensional state vector of the system, and

g is a smooth nonlinear function defined on ffi. D X R The output (observed) signal of the system s is generally given by s = rfJ(x(t)),

where

rfJO

(4.2)

is a smooth continuous scalar real-valued function. The goal is to

reconstruct the attractor using only a function of a subset of the state variables. In the Euclidean space ffi.M, the chaotic attractor of Eq. (4.l) can be reconstructed

from

s=[s 2

s s ... dM-l

S(M-l)f,

ds d s . s s,. .s, ... , s (M-l) denote - , - 2 ,. .. , ｾＬ＠ dtdt dt

where

ｍｾＨＲｄＫｬＩＬ＠

and

. 1y [ 19]. In other words, respectlve

Chapter 4

there exists a function

I]'

Signal Reconstruction in Noisefree and Distortionless Channels

such tha x = 1]'(S,t),

(4.3)

where 1]'=['1/1 'l/2···'l/D]T.It should be noted that s=¢J(x(t)) Vx¢J(x(t)).g(x,t)

def

def

J;(x,t),s=

12(x,t),and ds(i-I)

s(i)=-dt =

VxJ;(x,t). g(x,t) + oJ;(x,t)

at

(4.4) whereVxJ;(x,t) is the gradient ofJ;(x,t) with respect to x,and "." denotes the vector dot (inner) product. Also,

s = f(x,t), where f

= [J; h ... 1M t ,and

(4.5)

J; (i = 1,2" . " M) is a smooth function. Combining

Eq. (4.3) and Eq. (4.5), we have x = I]'(f(x,t),t).

(4.6)

By taking the derivative with respect to t of S(M-l) = 1M (x, t) and using the above equations, we obtain

(4.7) Defining the variablesYI =S'Y2 =S'Y3 =s,""YM =S(M-ll, we have another form ofEq. (4.1) in a higher dimensional space: Yl=Y2'

Y2 = Y3' YM

ｾ＠

1

(4.8)

h(Yph,···,YM,t)· 61

4.2

Reconstruction and Observability

Suppose that h is a polynomial that does not depend on t. Then we can determine the parameters of h. The function h can be written as:

(4.9) The parameters ak\,k"k3 , .. ,kM can be determined from set) by finding sand S(M)

at a finite number of time points and plugging into Eq. (4.9) to obtain a

set of linear equations which we can solve if s has not degenerate. Thus, it is generally possible to reconstruct the chaotic attractor of Eq. (4.1) on the higher-dimensional space from given s(t). It is also possible to estimate unknown parameters using the above method. CD The requirement of M? 2D + I is a suffcient condition on the dimension of the reconstructed phase space for this method to work. In certain cases the attractor can be reconstructed using a smaller M. Specifically, we consider the reconstruction of a class of Lure system in the next section, where M can be set to D.

4.2

Reconstruction and Observability

The pair (A, w

T

)

is said to be observable if the matrix

wT A=A(A,WT)=

wTA wTA2

(4.10)

is nonsingular, where A is a D x D matrix and w is a D x I vector. In

linear

system

eD = [0 0 ...

If E jRD,

theory,

the

statement

that

ＨａＬ･ｾＩ＠

where

is observable means that we can reconstruct the state

vector x in the system governed by the dynamics

x =Ax

via observing

CD The same conclusion holds for a chaotic system with a hysteresis loop if the system can be considered to be smooth.

62

Chapter 4

Signal Reconstruction in Noisefree and Distortionless Channels

only xD for some time [210]. We consider now the nonlinear Lure system defined by (4.11 )

Of E]RD,

where el = [1 0 0

U

is a continuous real-valued function, and

A is given by 0

0

-bo

1 0

0

-bl

0

0

-b2

0

0 where bi, i

=

(4.12)

1 -bD _ l

0

0, 1, 2,"', (D -1), is the coefficient of the characteristic

polynomial of matrix A, i.e., (4.13) When ＨａＬ･ｾＩ＠

is observable, we can reconstruct the system dynamics via

(XD,XD,X D,.·

·,xb

D l - )).

In fact, if we defmec( as the ith row of Aj, it is easy to

see thatcbel =0 for j=O, 1,.··,(D-2). Then, x D ］･ｾｸ｣Ｌｘｄ＠ ｸｄ］｣ＱａＫ･ｬｵＨＬｴＩｾＮﾷＭ＠ ｛･ｾｸ＠

=c1x, and

This c1x ... ｃｾＭｬＩ＠

xf = ａＨＬ･ｾＩｸＮ＠

an invertible linear mapping between

ｓｩｮ｣･ａＨＬｾＩ＠

means

that

s=

is invertible, there exists

s sand x. We can write that Xl

as (4.14)

Therefore, we can reconstruct the system dynamics of the Lure system.

4.3

Communications Based on Reconstruction Approach

Consider the well-known Duffing's equations (4.15)

63

4.3

Communications Based on Reconstruction Approach

where

rj

and

r2

are some constant parameters. Shown in Fig. 4.1 is the

two-dimensional phase space trajectory of the Duffing's equations with rj

= 0.15 and r2 = 0.3. 1.5

1.0

0.5

0

>< -0.5

-1.0

-1.5

-1.5

-1.0

-0.5

0

0.5

1.0

1.5

XI

Figure 4.1 The two-dimensional phase space trajectory of the Duffing equations with the parameter values rl

= 0.15 and r2 = 0.3

The above equations are equivalent to the following scalar differential equation (4.16) Therefore, if the signal Xj(t) is given, we can easily reconstruct a chaotic attractor ofEq. (4.15), by using

(XI (t),

Xl (t)) , due to the following relation:

(4.17)

i.e., the system being observable.

4.3.1

Parameter Estimations

The parameters in Eq. (4.15) can be estimated from a given signal. If Xj (t) is 64

Chapter 4

Signal Reconstruction in Noisefree and Distortionless Channels

given, then we can find the parameters rl and r2 from the following relation: (ll) - COS(ll) ] ['i 1 [Xl (ll) - X;3 (ll) - ｾ＠ (ll) ]. ｛ Xlｾｬ＠ (l2) - COS(l2) r2 XJ(2) - Xl (t2) - Xl (lJ =

Shown in Figs. 4.2 and 4.3 are the errors of the estimated

(4.18) r]

and

r2

as a

-76

-77

S ｾ＠

,,"

"0

'" Ｎｾ＠

"0

'-

g

P.l

-S2 -S3

0

100

200

300

400

500

Number of timesteps

Figure 4.2 Error curve of the estimated system parameter rl versus the time step -]07

-lOS

iii: ｾ＠

"0

Ｎｾ＠

'"

'"

-III

'"0 '....0 t:

P.l

-113 -114

0

100

200

300

400

500

Number of time steps

Figure 4.3 Error curve of the estimated system parameter r2 versus the timestep

65

4.3

Communications Based on Reconstruction Approach

function of the timestep, when the real parameter values in Eq. (4.15) are set to

rl

= 0.15 and r2 = 0.3.

4.3.2

Information Retrievals

An information signal v(t) can be injected into the system Eq. (4.15) (4.19) or equivalently

X1 =x2 ' x2 = -fix2 + x 1 - ｸｾ＠

(4.20)

+ r2 cos(t) + vet).

Then, vet) can be reconstructed by using (X1 (t),X1 (t),x1(t)): (4.21 ) Figure 4.4 shows the retrieved information signal (circled line) when vet) to be injected into Eq. (4.15) is a sinusoidal function (solid line) of frequency 1/ 811: and amplitude 0.1. 0.15

o

0.10

0.05

s

0

'" -0.05

-0.10 0

Original Retrieved

AI ' J

f\\

\ ｾ＠ 20

\

\\ t \;

40

60

80

100

Time

Figure 4.4 The retrieved information signal and the original one injected into Eq. (4.20)

66

Chapter 4

Signal Reconstruction in Noisefree and Distortionless Channels

Another system to be considered here is the Chua's circuit studied in Chapter 2. Let us rewrite the dynamical equations as follows

(4.22)

X2 =xI -X2 +X3' X3 = a l x 2 ,

where K(-) is a piecewise-linear function as described in Eq. (2.8). We can reco-

x

nstruct the chaotic attractor by using (X3' 3' x 3) when a2 -:t o. In fact, we have

X3

=

a 2 x2 ,

X3

=

a 2(xl -X2 + xJ.

(4.23)

Thus, we obtain

(4.24)

Furthermore, putting Eq. (4.23) and Eq. (4.24) in Eq. (4.22), we have (4.25) If K(·) is a cubic polynomial K(X) = d 1X3+ d2x [138], then we get

a;xi3) + dial (x; + Ｓｘｾ＠ - 6a2x3x3x 3 + Ｓ｡［ｸｾ＠ ··2 X3 - 3a 2x··23x3 +X3·3 - 3a 2x·23x3 + 3a 22·x3x 32 -a23 x33) + 3X3 + a; [(1 + d 2a l )x3 + (-a 2 -al +d2a l )x3 (4.26)

+dpI(al -a2)x3]=O.

We can estimate the parameters in Eq. (4.22) from the given signal X3 by using (X3 (t), X3 (t), X3 (t), xi 3)(t)). Furthermore, we inject the information signal vet) into the system:

J- ()

•. X3(3) +X3 -(al +a2)x. 3 +ap2 K [X3 +X3a -a2X3 -a2v t , 2

(4.27)

67

4.3 Communications Based on Reconstruction Approach

or equivalently

(4.28)

Xl =Xj -Xl +X3'

X3 = alxl ·

For a l (x],

-:f-

0 we can reconstruct (x\,

X2, X3).

X2, X3)

from

X3

and then recover vet) from

In other words, the chaotic attractor and v(t) are reconstructed by

(4.30)

Figure 4.5 shows the retrieved information signal (circled line) when vet) to -

0.25

o

Original Retrieved

0.20 0.15 0.10

S

"

0.05 0

1\ \

. I

I

\f

1\ \

-0.05 -0.10 -0.15 -0.20 0

5

10

15

20 Time

25

30

35

40

Figure 4.5 The retrieved information signal and the original one is injected into Eq. (4.28) with a j = 9 and a 2 = -100/7

68

Chapter 4 Signal Reconstruction in Noisefree and Distortionless Channels

be injected into Eq. (4.28) with

a]

= 9 and

a2

100

= -""7' is a sinusoidal

function (solid line) of frequency lin and amplitude 0.2.

4.4

Reconstruction of Attractor for Discrete Time-Varying Systems

The reconstruction approach can be applied not only to continuous-time dynamical systems, but also to discrete-time dynamical systems. Only a slight change is needed for application to discrete-time systems. The results obtained for continuous systems are transformed to those for discrete-time dynamical systems by replacing the derivatives with time-advanced state variables: (x(t), x(t),x(t),"') ｾ＠

(x(t),x(t + 1), x(t + 2)", J

(4.31 )

The advantage of using discrete-time systems is that there are no errors introduced by numerical differentiation, unlike in the continuous-time case. Consider the HEmon map governed by the dynamical equations: x\(t + 1) = 1- ｲ｜ｸｾＨｴＩ＠

+ x2(t),

(4.32)

x 2(t + 1) = r2x\ (t),

where r\ and r2 are the two bifurcation parameters of the system, which determine the dynamical behavior of the system(]).For r2

-:j:.

0, the attractor

can be reconstructed from X2(t) and X2 (t + 1) as

The information signal vet) is then injected into the system

CD For some selected range of parameter values, the system can show the chaotic behavior (see Sec. 4.2).

69

4.4

Reconstruction of Attractor for Discrete Time-Varying Systems

XI (t + 1) = 1- rlxlz (t) + Xz (t) + vet), X z (t

(4.33)

+ 1) = rzxI (t),

or equivalently (4.34) Shown in Fig. 4.6 is the chaotic atiractor of the Henon map when the system parameters are Yl = 1.4 and Y2 = 0.3 and the inj ected signal is a sinusoidal function of amplitude 0.001 and frequency 1/20 . 0.4 0.3 0.2 0.1

k'

0 -0.1 -0.2 ..•....

-0.3 -0.4 -1.5

-1.0

-0.5

0.5

1.0

1.5

Figure 4.6 Chaotic attractor of the Henon map with YI = 1.4 and Yz = 0.3. The

injected signal vet) in Eq. (4.33) is a sinusoidal function of amplitude 0.001 and period 1/20

Again, when Y2;j:. 0, set) can be recovered from X2(t), X2(t +1), and

x2(t+2) as: vet)

X

z

(t + 2) rz

ＫＭｸｾＨｴｬＩｺＱＮ＠

r

(4.35)

rz

Shown in Fig. 4.7 are the recovered information signal (circled line) and the original one (solid line) injected into Eq. (4.33).

70

Chapter 4

Signal Reconstruction in Noisefree and Distortionless Channels

-Original o Retrieved

0.5

S

0

;:.

-0.5

-\.O 0

25

50

75

100

125

ISO

Time Figure 4.7 The retrieved information signal and the original one injected into

Eq. (4.33)

4.5

Summary

In this chapter, we have extended the Takens' embedding theory to the reconstruction of continuous and discrete-time varying systems based on the observer approach. In particular, we have studied the Lure system, which can be reconstructed in a state space whose dimension is equal to the degree of freedom of the system. Also, the information signals injected into the so-constructed chaos-based communication systems in the absence of noise and distortion can be retrieved.

71

Chapter 5

5.1 5.1.1

Signal Reconstruction from a Filtering Viewpoint: Theory

The Kalman Filter and Extended Kalman Filter The Kalman Filter

The Kalman filter (KF) is a recursIve filtering tool which has been developed for estimating the trajectory of a system from a series of noisy and/or incomplete observations of the system's state [211- 231]. It has the following characteristics. First, the estimation process is fonnulated in the system's state space; Second, the solution is obtained by recursive computation; Third, it uses an adaptive algorithm, which can be directly applied to stationary and non-stationary environment [214]. In the Kalman filtering algorithm, every new estimate of the state is retrieved from the previous one and the new input so that only the previous estimated result need to be stored, and all those before the previous one can be discarded. Thus, the Kalman filter is more effective in computation than those which use all or considerable amount of the previous data directly in each estimation [214, 215]. For the purpose of our discussion, it suffces to consider the state of a system as the minimum collection of data which can describe the dynamical behavior of the system. Knowledge of the state is necessary for the prediction of the system's future trajectory, and is also relevant to the past trajectory of the system. In discrete time, a dynamical system can be described by a process equation that essentially defines the system dynamics in tenns of the state, and an observation equation that gives the observed signal, i.e.,

Chapter 5

Signal Reconstruction from a Filtering Viewpoint: Theory

• Pcrocess equation The process equation of a dynamical system can be generally written as (5.1) where xn is an N-dim state vector of the system at discrete time n; F is an N x N state transition matrix representing the movement of the dynamical system from time n to n + 1 and is usually given; wn is an N-dim process noise vector that describes the additive noise modelled as a zero-mean, white noise process whose correlation matrix is defined as: (5.2) where

Q;

is a diagonal matrix and 0 is a zero matrix.

• Measurement equation The measurement equation provides a linkage between the observed output and the system's state, and can be described as (5.3) where Y n is an M-dim vector representing the observed or measured output of the dynamical system at instant n;

en

is an M x N observation matrix,

which makes the state observable and is usually required to be given; vn is an M-dim observation noise vector, which is modelled as a zero-mean white noise process whose correlation matrix is defined as:

{Q0,n' V

E[vnvJ] = where

Q;

n- k -

n"#

k

(5.4)

is a diagonal matrix and 0 is a zero matrix.

Suppose wn and vn are statistically independent. Then, we have (5.5)

73

5.1

The Kalman Filter and Extended Kalman Filter

Therefore, the filtering problem, namely solving the state process equation and the observation equation by means of optimization, can be described as follows. From the observed dataYpY2" .. ,Yn' for all n"? 1, the problem is to find the least squared estimate of each component of the state variable Xi' If

n = i, the problem is a filtering one; if i > n , it belongs to prediction; and if 1 ,,;; i < n, , then it would be a process of smoothing.

We introduce an innovation process (innovation in short) to Kalman filtering, which represents the update of new information of the observation vector yen). In effect, innovation can be regarded as one-step forward prediction. With the observed values yp Y2' ... , Yn-P we can obtain the least A f0 b ' servatIon vector Y n as YAdef n=Yn IYp ... ,Yn- 1• So the M-dim innovated vector, an , is generally given as

· squared estImate

0

(5.6) Note that an is a Gaussian noise process which is uncorrelated with the observed data YP""Y n-1 before the instant n [219]. Nonetheless, it provides new information about Y n • This is why we name it as innovation. Basically, the Kalman filter algorithm includes two stages: prediction and correction. We define the one-step predictive value and the estimate of the state vector, respectively, as (5.7) (5.8) A summary of the main parameters and equations in the Kalman filter algorithm is as follows.

• One-step prediction of initial state Correlation matrix of predicted state errors:

74

Chapter 5

Signal Reconstruction from a Filtering Viewpoint: Theory

• Input vector Observed vector sequence: {YPY2, ... ,Yn}

• Given parameters State transfer matrix: F(n + 1,n) Observation matrix: Cn Correlation matrix of process noise vector:

Q:

Correlation matrix of observation noise vector:

Q;

• Prediction One-step state prediction

xn : Xn =F(n,n-1)xn_1

One-step prediction for observed vector Yn

(5.9)

:

Yn =CnXn

(5.10)

A

Correlation matrix of state prediction error Pn1n -1 :

Pn1n -1 = F(n,n -1)Pn_1F(n,n-1) +

Q:

(5.11)

• Error correction Kalman gain matrix Kn : (5.12) Innovation a(n) : (5.13) Error correction for state estimate

xn: (5.14)

Error correction for correlation matrix of state estimate errors: (5.15)

75

5.1

The Kalman Filter and Extended Kalman Filter

In conclusion, the Kalman filter is essentially a finite-dimensional linear discrete-time recursive algorithm. The core of the algorithm is to minimize the trace of correlation matrix of the state estimate error Pn • In other words, the Kalman filter performs a linear least squared estimation for the state vector xn [221, 223].

5.1.2

Extended Kalman Filter

The Kalman filter algorithm is a stochastic estimation tool for linear systems. If the system is nonlinear, the Kalman filter cannot be applied directly, and an additional linearization process is required. Incorporating a linearization process in the Kalman filter, we have the extended Kalman filter (EKF) [232 - 247]. Consider a nonlinear system whose state-space description is given as (S.16) (S.17)

where wn and vn are uncorrelated zero-mean white nOIse processes with correlation matrix Q: and ｑｾＬ＠

respectively, and F(n, xJ is a nonlinear

transfer function that may vary with time. Moreover, if F is a linear function, we have (S.18)

The fundamental idea of the EKF lies in linearizing the state-space model given in the form of Eqs. (S.16) and (S.17) at each time instant by using a first-order Taylor's expansion. Once the linearized system is obtained, the Kalman filter algorithm described in the previous section can be applied. The linearization process is described mathematically as follows: First, we construct two matrices as follows. F(n + 1,n) = of(n,x)1 ox ｘｾＢ＠

76

'

(S.19)

Chapter 5

Signal Reconstruction from a Filtering Viewpoint: Theory

(5.20) Next, we obtain F(n+ l,n)and C(n) by using F(n,x n ) and C(n,x n ) , and expand them according to first-order Taylor expansion at

x

n

•

This gives (5.21) (5.22)

Then, we linearize the nonlinear state-space representation (i.e., Eq. (5.16) and Eq. (5.17)), with the help ofEqs. (5.21) and (5.22). Finally, we obtain (5.23) (5.24) which correspond to Eq. (5.9) and Eq. (5.13) in the basic Kalman filter formulation. The complete EKF algorithm can be formulated by putting the linearized terms described above into the Kalman filter algorithm (from Eqs. (5.9 - 5.15)). It should be noted that the EKF algorithm is, in fact, a direct application

of the original Kalman filter algorithm with an additional linearization process. Moreover, since the EKF uses a first-order approximation, when the system has a higher degree of nonlinearity, the error caused by the higherorder terms of the Taylor's expansion might cause the algorithm to diverge [242,245,247], as will be demonstrated in the simulation experiments to be reported in the next chapter.

5.2

The Unscented Kalman Filter

Recently, a new type of filters, known as the unscented Kalman filter (UKF), has been proposed for noise cleaning applications [248,249,250]. The fundamental difference between EKF and UKF lies in the way in which 77

5.2

The Unscented Kalman Filter

Gaussian random variables (GRV) are represented in the process of propagating through the system dynamics. Basically, the UKF captures the posterior mean and covariance of GRV accurately to the third order (in terms of Taylor series expansion) for any form of nonlinearity, whereas the EKF only achieves first-order accuracy. Moreover, since no explicit Jacobian or Hession calculations are necessary in the UKF algorithm, the computational complexity of UKF is comparable to EKF. Very recently, Feng and Xie [260] applied the UKF algorithm to filter noisy chaotic signals and equalize blind channels for chaos-based communication systems. The results indicate that the UKF algorithm outperforms conventional adaptive filter algorithms including the EKF algorithm. In this section, we evaluate the performance of UKF in filtering chaotic signals generated from one-dimensional discrete-time dynamical systems which basically include most behaviors and characteristics in many multidimensional systems, and are widely used in signal processing and communications with chaos.

5.2.1

The Unscented Kalman Filtering Algorithm

Consider a nonlinear dynamical system, which is given by (5.25) where f: lR N ｾ＠

lR N is a smooth function, and the measurement equation is

given by (5.26) where

vn

is a zero-mean white Gaussian noise process, E

[VjVn]

= Qbjn

> 0,

and b jn is the Kronecker delta function, Q is a matrix with a suitable dimensionality. When the global Lyapunov exponent of Eq. (5.25) is positive, the system is chaotic. We will employ the the UKF algorithm

78

Chapter 5

Signal Reconstruction from a Filtering Viewpoint: Theory

developed in [248,249,250] to filter the noisy chaotic time series, i.e., to estimate the state xn fromy n . Assume that the statistics of random variable x (dimension L) has mean x and covariance Px • To calculate the statistics of y, we form a matrix X of 2L + 1 sigma vector Xi according to the following:

Xo

(5.27)

=X

Xi ］ｸＫＨｾｌａＩｐｩＧ＠ Xi

= x ＭＨｾｌ＠

i=I,2,. .. ,L

(5.28)

+ A)Px )i-L' i = L + 1,L + 2,. .. ,2L

(5.29)

where A = a 2 (L + K) - L is a scaling parameter. The constant a determines the spread of the sigma points around x and is usually set to a small positive value (0.0001 < a < 1). The constant K is a secondary scaling parameter which is usually set to Also, Ｈｾｌ＠

+ A)PX)i

def

K

= 0 (for parameter estimation,

K

=3 -

L).

ei is the vector of the ith column of the matrix square

root. These sigma vectors are propagated through the nonlinear function (5.30) The mean and the covariance of z can be approximated by using the weighted sample mean and covariance of the posterior sigma points, i.e., (5.31 ) i=O

2L

P"z = LWY)(z-z)(z-zY

(5.32)

ｩｾｏ＠

where weights

Wi

are given by m

A

(5.33)

w =-o L+A

ｷｾ＠

A L+A

= - - + (1- a

2

+ /3)

(5.34)

79

5.2

The Unscented Kalman Filter

wm

=

= ｷｾ＠

I

I

1 . , 1 = 1,2, .. ·,2L 2(L+A)

(5.35)

f3 is used to incorporate prior knowledge of the distribution of x Gaussian distributions, f3 = 2 is optimal) [248, 249].

where

(for

The transform process described above is known as unscented transform (UT). Now the algorithm can be generalized as follows:

Time update: (5.36) (5.37) 2L

ｾｉｮＭｬ＠

=

L w; (Xi,nln-l - xn1n - )(Xi.nln-l - xn1n 1

(5.38)

1) T

ｩｾｏ＠

2L

Px, y, =

L w; Cri,nln-l - xn1n - )(Yi,nln-l - Ynln- f 1

(5.39)

1

i=O

where ｾＭＱ＠

%nln-1'

xn1n -

1

and Pn1n - 1 is the predicted estimation for

%n-l'

x n_1 and

respectively. Px, y, is the covariance matrix of x and Y at instant n.

Measurement update: (5.40) (5.41) Pn =Pnln-l -Kn PYn Yn KTn

where Py ,

y,

is the covariance matrix of y at instant n,

(5.42)

Ynln-l

is the estimation

of the observed signal, and Kn is the Kalman gain matrix at time instant n. Using Eqs. (5.26), (5.32), (5.38) and (5.39), we can get Pnln-l =PXn Yn =PYn Yn -Qn'

80

(5.43)

Chapter 5

Signal Reconstruction from a Filtering Viewpoint: Theory

which then gives

P=KQ n n n

(5.44)

where Qn is the covariance matrix of the measurement noise at instant n. Furthermore, Pn1n -1 can be calculated as follows. Ifwe consider the prior variable x by a zero-mean disturbance ｾ＠

= x - X, then Px = ｾｸ＠

ｾｘｔＬ＠

ｾｸＬ＠

where

so that the Taylor series expansion of

Eq. (5.25) is given by

I(x) = I(x) + ｦＧＨｸＩｾＫ＠ where

0 (-)

ＮＡＨｾｸｦ＠

rex) + 001 ｾｸｉＱＲＩ＠

2

(5.45)

denotes infinite smallness, and II . II denotes the Euclidean norm.

From Eqs. (5.27 - 5.29), we know that Xn-l is symmetrically distributed L

around xn_l' Since ｾ＾ｩＮｮＭｬ＼＠

+ A )Pn-l' we can write x n1n _1 in Eqs. (5.37)

= (L ｩｾｏ＠

as

x n1n - 1 = ｾ＠

2L

m[

wi I(x n- l ) + f'(xn-l)(ei,n-l)

ＫｾＨ･ｩＬｮ｟ｉＩｔ＠

r(ei,n_I)+oOlei,n_lln]

(5.46)

Since 2L

2L

L w; = L w; + (1- a ｩｾｏ＠

2

+ /3)

(5.47)

ｩｾｏ＠

we can express Pn1n -1 as 2L

Pn1n -1 = "＠ｾ " Wim (Xi,nln-l

-

Xn1n -1)(Xi,nln-l A

-

Xn1n -1 )T A

i=O

2

+ (1- a + /3)(XO,nln-1 )T . XO,nln-1 - Xn1n - 1

-

Xn1n - 1 A

)

A

(

Again, Px

(5.48)

= E[(x - x)(x - X)T] = E(XXT)- xxT. We define

81

5.2

The Unscented Kalman Filter 2L

U=

L w;m (Z;,nln-I - xn1n - 1)(Z;,nln-I - xn1n _

l )T

(5.49)

i=O

V = (1- a 2 + fJ)(ZO,nln-1 We also define f' = F and

f" = H.

ｾＩＨ＠

X n1n _1

ZO,nln-1 -

ｾＩｔ＠

X n1n _1

(5.50)

Now, since

2L

ｌｗｾ＠

(5.51)

=1, ［ｾｏ＠

we get U=

ｆｮｾｉｐＬ｟Ｑ＠

Ｍｾ｛Ｈｈｮ｟ｉｐｙ＠

(Hn_IPn_I )]

+0(11 en _ 1 W)

(5.52)

(5,53) Since a is small, so is en_I' and the higher-order terms can be truncated. Thus, P n1n - 1 can be expressed as

(5.54) where F n _ 1 and H n _ 1 are the Jacobian and Hession matrices of Eq, (5,25), respectively.

5.2.2

Convergence Analysis for the UKF Algorithm

When we consider a one-dimensional system, Eq. (5.54) can be simplified as 2 2 Pnln-I =F2n-I Pn +'!"(fJ-a )Hn-I p2 4 n-I

82

(5.55)

Chapter 5

Signal Reconstruction from a Filtering Viewpoint: Theory

By using Eqs.(5.44), (5.54) and (5.55), we can calculate the Kalman gain as k

= n

l

F2 +l(p_a2)H2 p2 n-I

4

n-I n-I

]k

n-I

(5.56)

[F2n-I + 4(P 1 - a 2 )Hn-1P"_1 2 ] kn_ 1 + 1

Note that both p" and kn can be used to measure the filter's performance. However, since they are linearly dependent, as shown in Eq. (5.44), it suffces to analyze the behavior of any one of them. For simplicity, we focus on kn • In general, four types of behavior can be identified in a nonlinear dynamical system: • stable fixed point; • periodic motion; •

quasiperiodic motion;

• chaotic state. In a one-dimensional dissipative system, a quasi-periodic state corresponds to a zero Lyapunov exponent, and it is a critical state with zero measure of parameters. It is therefore not of interest to our present study. Here, we consider two classes of dynamical behavior, namely, chaotic and periodic, as the stable fixed point can be regarded as a special periodic state. Our analysis for the UKF algorithm used to filter a chaotic system is summarized in the following theorems. In particular, we consider three types of nonlinear dynamical systems which determine the behavior ofUKF. is independent of x n _ l • • Type 1: ｆｮｾＱ＠ • Type 2: ｆｮｾＱ＠

is dependent upon x n _ 1 , but

• Type 3:

is dependent upon x n _ l .

H;_I

H;_I

is independent of x n _ l •

Theorem 5.1 For Type 1 systems, the Kalman gain k n given in Eq. (5.56) converges to zero when the systems being dealt with are periodic, and converges to a non-zero fixed point when the systems are chaotic. Proof: As ｆｮｾＱ＠ is independent of X n _ I ' ｆｮｾＱ＠ can be regarded as a constant i;. Thus, Eq. (5.56) becomes 83

5.2

The Unscented Kalman Filter

k = n

qkn _ 1 def J:k +1

'='

do 'f/

(k

J:)

n-P':> ,

(5.57)

n-l

which includes two fixed points, i.e., (5.58) The stability of the two fixed points can be determined by the derivative of ¢ at the corresponding fixed points. If the derivative has a magnitude larger than one, the fixed point is unstable, and the Kalman gain cannot converge. Otherwise, the fixed point is stable. By evaluating the derivative ofEq. (5.57) at the two fixed points, we have (5.59) When q < 1, k n = 0 is a stable fixed point, and k n = (q -1) / q is an unstable fixed point. Conversely, when kn

=

(q -1) / q

q > l,kn

=0

is an unstable fixed point, and

is a stable fixed point.

The global Lyapunov exponent of the one-dimensional system can be written as

s = lim -N1 L In 1F

1

N

N->oo

n=1

n

1= -InC q). 2

(5.60)

Thus, when Eq. (5.25) is periodic, ¢'< 1, and kn converges to k(J) . Moreover, when Eq. (5.25) is chaotic, k n converges to Theorem 5.2 For Type 2 systems, if ｈｾ｟ｉ＠

k(2) •

= q oF-

0 (q is a constant), the

Kalman gain kn in Eq. (5.56) converges to zero when the systems being dealt with are periodic and k n oscillates aperiodically when the systems are chaotic. Proof: From Eq. (5.56), we obtain

84

Chapter 5

Signal Reconstruction from a Filtering Viewpoint: Theory

(5.61)

IfEq. (5.25) is chaotic and the square of the gradient Fn2 is aperiodic, then ｆｮｾｬ＠ + (114 )q(P - a 2) is also aperiodic. Hence, the result follows according to Theeorem 2 in [251]. If we define ･ｮ｟ｬｾ＠

= p"-p then Hn_le n_l = Fn - Fn_l . By using Eq. (5.56),

we can obtain

(5.62)

2 Lemma 1: Suppose Gn = Fn2_1 + (1/4)(p-a )(Fn -Fn_I)2. Then, Gn is aperiodic. Proof: Assuming that Gn has a period of N, without loss of generality, we have

Gn+N -Gn = 0.

(5.63)

Define

ml,n

= F;+N_1 - ｮｾｬ＠ｆ

2 2 m2.n = Fn +N - Fn I],n

= Fn+N

r2,n

= F'" - F',,-l

- Fn+N- 1

(5.64) (5.65) (5.66) (5.67)

and consider ｆｮｾｬ＠

being aperiodic. Then, when l],nFn+N - r2,nFn = 0, 2 Gn+N - Gn = m\,n + (114) (P - a ) (ml,n + m2,n - 2m 2,J is not always equal to 2 zero. Also, when 2 (l],nF',,+N-r2,nF',,) =In-:F-O,Gn+N-Gn =m\,n+(1I4)(p-a ) 85

5.2

The Unscented Kalman Filter

(mI,n + m2 ,n - 2m2 ,n + IJ is not always equal to zero. Therefore, the above

assumption about the periodicity of Gn is not valid. In other words, Gn is aperiodic. Theorem 5.3 For Type 3 systems, the Kalman gain k n in Eq. (5.56) converges to zero when the systems being dealt with are periodic, and k n oscillates aperiodically when the systems are chaotic. Proof: Since Gn is aperiodic, Type 3 systems can be arranged to Type 2 systems, and the result follows from Lemma 1.

5.2.3

Computer Simulations

To illustrate the theoretical results developed in the foregoing section, we consider three systems, i.e., the tent map and two logistic maps. 5.2.3.1

Type 1

We consider the following class oftent maps: Xn+I = a(1-12xn -11)

where a E [0, 1] and xn

E

(5.68)

[0, 1]. The global Lyapunov exponent and local

exponent of the system are identical, and equal to In 2a. The map is chaotic for a > .!.-, and is periodic otherwise. The UKF algorithm for this map can 2

be realized by first using Eq. (5.64) to get the Jacobian matrix, i.e., 2a,

1 x -. n 2

F= n

2

Then, we have Fn

=

4a

2 ,

which is independent of x n' According to

Theorem 1, the Kalman gain converges to

86

(5.69)

Chapter 5

Signal Reconstruction from a Filtering Viewpoint: Theory

limk = n-->oo

n

o,

1 a-.

4a 2 -1 4a 2

'

(5.70)

2

Figure 5.1 shows the typical convergence behavior of the Kalman gain for this map, in which the dotted lines denote the steady states given by Eq.(5.70). We can see from Fig. 5.1 that the magnitude of the fixed point increases as the Lyapunov exponent gets larger when the system is chaotic, as given in Theorem 5.1. 1

0.9 0.8 0.7

IV

0.6

"'c"

'0:;

0.5

01)

c

"E ""OJ ｾ＠

"

a=0.8

0.4 0.3 0.2 1

l / a =0.3

0.1 0

\a=O.4

-0.1 0

10

20

30 Iteration

40

50

60

Figure 5.1 Convergence characteristic of the Kalman gain for tent map with different

values of a corresponding to periodic signals (a = 0.3 and a = 0.4) and chaotic signals (a = 0.8 and a = 0.9). Dotted lines denote the theoretical steady states

5.2.3.2

Type 2

We consider the logistic map given by (5.71)

87

5.2

The Unscented Kalman Filter

where

aE

[0,4] and

Xn E

[0, 1]. The Jacabian matrix of the map is

Fn =

a - 2axn , and the Hession matrx is - 2a. Thus, this map is a Type 2 system.

Figure 5.2 shows the convergence behavior of kn' in which the dotted, dashed, and solid lines describe the kn 's convergence behavior for the system with parameter a= 3.4, a= 3.84 and a= 4.0, respectively. The former two are periodic, while the latter gives a chaotic time-series. When the timeseries is periodic, k n converges to the fixed zero state. However, when the time series is chaotic, k n exhibits an aperiodic behavior, as described in Theorem 5.2. a=3.4 - -- a=3.84 a=4.0

0.9 0.8

1\

A

A

A

"'c"

ｾ＠

1\

0.7

n

Ii

0.6

'0;;

bIJ

c 0.5 co

E

(ii

::.c 0.4 0.3 0.2 0.1

'i

. ,t" 1\

o o

j

l,' '\ 20

40

60

80

100

Iteration

Figure 5.2 Convergence characteristic of the Kalman gain of the logistic map

(Type 2). Dotted, dashed, and solid lines describes the k n 's convergence behavior for the system with parameter a=3.4 (periodic), a=3.84 (periodic), and a=4.0 (chaotic), respectively

5.2.3.3

Type 3

We consider another form oflogistic map, which is defined as (5.72) 88

Chapter 5

where a E [0, 1.45] and xn

Signal Reconstruction from a Filtering Viewpoint: Theory

1.5, 1.5]. The Jacobian matrix of the map is Fn = an cos (nXn) , and the Hession matrix is Hn = -an2 sin (nxJ, which E [ -

are dependent on x n • This map is therefore a Type 3 system. Figure 5.3 shows the convergence behavior of kn' in which the dotted, dashed, and solid lines correspond to the results of a=

a=

0.8,

a=

0.94 and

1.0, respectively. The former two are periodic, while the latter gives a

chaotic time series. When the time series is periodic, k n converges to the fixed zero state. However, when the time-series is chaotic, k n exhibits an aperiodic behavior, as described in Theorem 5.3.

0.9 0.8

- _. a=0.8 a=0.94 a=1.0 -

(' ( n ｾ＠

0.7 0.6 Ｎｾ＠

c

0.5

B

E 0.4

" ::L

0.3 0.2 0.1

o ｾＭ

-0.05 L-_-'--_--'--_-'-_----'_ _'---_-'--_--'---_---'--_----'_ _ o 10 20 30 40 50 60 70 80 90 100

Iteration Figure 5.3 Convergence characteristic of the Kalman gain for the logistic map (Type 3). Dotted, dashed and solid lines describes k n 's behavior for the system with parameter a == 0.8 (periodic), a == 0.94 (periodic) and a == 1.0 (chaotic), respectively

5.3

Summary

In this chapter, we revisited the KF, EKF and UKF algorithms. In particular, the problem for filtering noisy signals arising from chaotic systems using the 89

5.3

Summary

UKF algorithm has been investigated. It has been shown that when a nonlinear system is chaotic, the Kalman gain of the UKF does not converge to zero, but either converges to a fixed point with magnitude larger than zero or oscillates aperiodically. The dynamical behavior of the error covariance matrix can be readily found since it is linearly related to the Kalman gain.

90

Chapter 6

6.1

Signal Reconstruction from a Filtering Viewpoint: Application

Introduction

As mentioned previously, separating a deterministic signal from noise or reducing noise for a noisy corrupted signal is a central problem in signal processing and communication. Conventional methods such as filtering make use of the differences between the spectra of the signal and noise to separate them, or to reduce noise. Most often the noise and the signal do not occupy distinct frequency bands, but the noise energy is distributed over a large frequency interval, while the signal energy is concentrated in a small frequency band. Therefore, applying a filter whose output retains only the signal frequency band reduces the noise considerably. When the signal and noise share the same frequency band, the conventional spectrum-based methods are no longer applicable. Indeed, chaotic signals in the time domain are neither periodic nor quasi-periodic, and are unpredictable in the long-term. This long-term unpredictability manifests itself in the frequency domain as a wide "noiselike" power spectrum [2, 151]. Conventional techniques used to process classic deterministic signals will not be applicable in this case. A number of noise reduction methods for noisy chaotic time series have been proposed [252 - 254]. These methods usually treat noise reduction for noisy delay-embedded scalar time series in a reconstruction space based on the embedding theory [1], or for observed noisy time series by using some filtering technique to obtain an optimal estimation of the original dynamical system [101, 255 - 257]. A popular technique for filtering noisy chaotic signals is to use an extended Kalman filter (EKF) algorithm, as described in the previous chapter. However, the side effects from using EKF for filtering

6.2

Filtering of Noisy Chaotic Signals

chaotic signals are lowered estimation precision due to the first-order approximation and higher complexity due to the need for computing the Jacabian matrix [178,258]. In this chapter, we describe a new method for filtering noisy chaotic signals and retrieving the original dynamics of a chaotic system.

6.2

Filtering of Noisy Chaotic Signals

6.2.1

Filtering Algorithm

Figure 6.1 shows a block diagram of the problem of filtering a noisy chaotic signal. Suppose the dynamics of the chaos generator can be described in the state space as (6.1) (6.2) where f: lR N ｾ＠

lR N is the nonlinear (chaotic) function, xn

E

lR N is the state

vector of the system at time instant n, xn is the output (scalar) signal of the chaos generator, Yn is the observed signal, and

Vn

is the measurement noise.

By using a filter, we can, from Y n , get the estimation of the source signal x n ' l.e., x n '

Figure 6.1 Block diagram of filtering a noisy chaotic signal

In estimating the state, the EKF algorithm is widely used to achieve a recursive maximum-likelihood estimation of the state xn [178,259]. The basic idea of the algorithm is summarized as follows. Given a noisy observation Y n , a recursive estimation for 92

xn

can be expressed as

Chapter 6

Signal Reconstruction from a Filtering Viewpoint: Application

(6.3) where k is the Kalman gain vector. The recursion provides the optimal minimal mean squared error (MSE) estimation for xn if the prior estimation

xn

and current observation Yn, are Gaussian random variables. In the

algorithm, each state distribution is approximated by a Gaussian random variable, which is propagated analytically through the first-order linearized model of the nonlinear system. This approximation, however, can produce large errors in the true posterior mean and covariance of the transformed random variables, which may even lead to the divergence of the algorithm [ 178]. Recently, the UKF has been proposed for solving the filtering problem [248,249], [260 - 262]. When the parameter of Eq. (6.1) varies with time, Eq. (6.1) can be rewritten as (6.4) where () is the parameter, which can be modeled by using the autoregressive (AR) model as [178] p

()n = L>J()n-J

+.9n

(6.5)

ｊｾＱ＠

where a.) is the coeffcient of the AR model, p is the order of the AR model, and .9n is additive white Gaussian noise. The problem is now changed into the estimation of the mixed state and parameter in the state space. Essentially, the augmented state-space vector is [xn ()n ＨＩｾＲ＠ ... Ｉｾｰ＠Ｈ an ｾＲＩ＠｡ ... ｡ｾｰＩ＠

r.The UKF algorithm can then be applied to this augmented state-

space model. The general algorithm for estimating the system can be summarized in two main stages.

Prediction Stage The purpose of the prediction stage is to calculate the following:

93

6.2

Filtering of Noisy Chaotic Signals

Xnln-1 =

Xnln-l

(6.6)

1(1'.) H

= "W(m)X

L....;

(6.7)

;,nln-l

i=O

H

Px,nln-l = "L.... w;(c) (X;,nln-l

')(

- X n1n - 1

')T

(6.8)

X;,nln-l - X n1n- 1

i=O

where

X nln-l

is the estimation for

Xn

at time instant n,

xn1n -

1

is the

estimation for x's at instant n, and Px ,nln-l is the estimation for Px at instant n.

Correction Stage In the correction stage, we perform the following calculations: 1 k=P-'Y py

(6.9) (6.10)

P=P n nln-l -kPk n y n

(6.11)

where P-'Y is the covariance matrix ofx andy, Py and

Ynln-l

are the variance

and the estimation ofthe observed signal. It should be noted that it is unnecessary to perform explicit computation

of the Jacobian matrix (partial derivatives) in this algorithm. In addition, the complexity of the UKF is, in general, O( A3 ) , where A is the dimension of the state [178, 248, 249].

6.2.2

Computer Simulation

In performing simulation, the output signal

Xn

of the chaotic generator is

controlled to reach the required signal-to-noise ratio (SNR) in a A WGN channel. For the retrieved signal Xn , the MSE is used to evaluate the algorithm's performance. The MSE is defined as:

94

Chapter 6

Signal Reconstruction from a Filtering Viewpoint: Application

(6.12) where Mis the number of the retrieved sample points. Three chaos generators, namely, logistic map, HEmon-map and Chua's circuit, will be used to demonstrate the performance of the algorithm in this chapter. The logistic map is described by the following equation (6.l3)

where the parameter

Ii (Ii E

(0,4]) is fixed at 3.8, which ensures the chaotic

motion ofthe system. Figure 6.2 shows the retrieved return map ofEq. (6.13) from xn with an SNR of 30 dB. Figure 6.3 shows the MSE of the retrieved signal versus SNR, in which the solid line and the dotted line correspond to the results by using UKF and EKF, respectively. It can be seen from Fig. 6.3 that the MSE by usingUKF is approximately - 45 dB when SNR is 30 dB, which slightly outperforms the EKF algorithm.

0.8 0.6 0.4 0.2 ｾ＠

.j.

I·

0

I

-0.2 -0.4 -0.6 -0.8 -1

I

-1

-0.5

0 Xn-l

Figure 6.2 Return map of the logistic map retrieved from noisy data (SNR= 30 dB)

by using the UKF algorithm

95

6.2

Filtering of Noisy Chaotic Signals

-10

-20

I--EKFI -UKF

,

-30 -40

EO -50

::s, ｾ＠

2

-60 -70 -80 -90 -100

10

20

30

40

50

60

70

80

SNR (dB)

Figure 6.3 MSE of the retrieved signal versus SNR for the logistic map (solid line

for UKF and dotted line for EKF)

When c varies with time, e.g., (6.14) we should model the system by using the second AR model as described previously and expand the system into the augmented state-space model. Then, the UKF algorithm can be used to estimate the states of the augmented model. Figure 6.4 shows the MSE of the retrieved signal xn versus SNR, in which the solid line and the dotted line represent the results by using UKF and EKF, respectively. Figure 6.5 shows the MSE of the retrieved parameter

cn versus SNR. We can see from Figs. 6.4 and 6.5 that the two filtering methods have the same performance when SNR is high, and the UKF outperforms the EKF when SNR is low. Next we examine a chaotic modulation system as described in [101], in which the logistic map Eq. (6.13) is again used as the chaos generator, and the value of each pixel point of the standard Lenna portrait (256 x 256 pixels) shown in Fig. 6.6 is used as the modulating signal. The modulation signal is

96

Chapter 6 Signal Reconstruction from a Filtering Viewpoint: Application

-10 -20

1--

EKFI -UKF

,

-30 -40

ill ｾＭＵＰ＠

w

ifJ

ｾ＠

-60 -70 -80 -90 10

20

30

40

50

60

70

80

SNR (dB)

Figure 6.4 MSE of the retrieved signal xn versus SNR for the logistic map when li varies with time according to Eq. (6.14) (solid line for UKF and dotted line for EKF) 10 5 0

ill -5 ｾ＠

ｾＭＱｏ＠

2 -15 -20 -25

UKF

10

20

30

40

50

60

70

80

SNR (dB)

Figure 6.5 MSE of the retrieved parameter li versus SNR for the logistic map when li varies with time according to Eq. (6.14)

injected into the system according to the following equation: lin =

3.57 + ¢(i,j)

(6.15)

where Ifi(i, j) is the pixel value of point (i, j). The quality of the retrieved portrait is evaluated by peak signa1-to-noise ratio (PSNR), which is defined as 97

6.2

Filtering of Noisy Chaotic Signals

PSNR(dB) = 1010glO(2552 I E)

(6.16)

where 256

E=

L

[r/J(i,j)-¢(i,j)]2/256

2

(6.17)

i,j=1,.··,256

and r/J(i,j) is the estimated value of r/J(i,j). When the SNR ofthe received signal xn is 50 dB, the PSNR of the retrieved portrait (shown in Fig. 6.7) using the UKF algorithm is 35.1 dB, which shows a 7 dB improvement in performance compared to that using the EKF algorithm.

Figure 6.6 The standard Lenna portrait with 256 x 256 pixels

Figure 6.7 The retrieved Lenna portrait

98

Chapter 6

Signal Reconstruction from a Filtering Viewpoint: Application

The second system we study here is the Henon-map, which is described by the following 2-dim iterative map: (6.18) (6.19) where a l is a time-varying bifurcation parameter (al

E

(1.32, 1.42]) whose

variation ensures chaotic motion of the system, and a 2 is fixed at 0.3. When

al

=

1.4, we evaluate the performance of the two algorithms. Figure 6.8

shows the MSE of the retrieved signal xn versus SNR, in which the solid line and the dotted line represent the results using UKF and EKF, respectively. We can see from Fig. 6.8 that the performance of UKF outperforms EKF, and the MSE of the retrieved signal using UKF is reduced by 17 dB when SNR is equal to 30 dB. 0 -10

1-EKF 1 - UKF

-

-20 -30

ｾ＠ -40 W ｾ＠

-50 -60 -70 -80 -90

10

20

40

30

50

60

70

80

SNR(d8)

Figure 6.8 MSE of the retrieved signal xn versus SNR for the Henon-map (solid

line for UKF and dotted line for EKF)

The third chaos generator to be used is the Chua's circuit, which is described by the following dimensionless equations [263]: x=

a 3(y - K(X))

(6.20) 99

6.2

Filtering of Noisy Chaotic Signals

(6.21)

y=x-y+z

(6.22) where a 3 is fixed at 9, and a 4 at -10017. This choice of parameters ensures the chaotic motion of the system. Also, KO is a piecewise linear function which is given by _{mjx+(mo-m j ), x;?l K(X)- mox, Ixl N. Similarly, for the cases where M < Nand M

=

N, the same conclusion

holds. The above estimation procedures, together with the training algorithm described in the following section, can be used to realize the equalization task.

147

8.2

8.2

Training Algorithm

Training Algorithm

Let d(t+l) be the desired output of the output unit at time instant t+1. The error signal e(t +1) is

e(t + 1) = d(t + 1) - x(t + 1).

(8.21)

The weight between the hidden layer and the output unit is then updated by a least-mean-square (LMS) algorithm [178], i.e.,

Ui(t + 1) = ui (t) + fJ1e(t + 1) qi(t + 1),

(8.22)

where fJI is the learning rate. The instantaneous sum of squared errors of the network is defined as &O(t+l)=(l/2)e 2 (t+l). Also, we defme the local gradient of the ith hidden unit at time instant t + 1, y; (t+ I), as

y(t+l)= ,

cl&°(t+l) Olj (t)

oq.(t + 1) = e(t + l)u i (t)--"-'-'-'--....:.. olj (I) =

e(1 + l)u i (t)tp'(lj (t)),

(8.23)

where tp' (-) is the derivative of tp with respect to its argument. According to the delta learning law, the weight

wij

(i = 1,2,.··, N, j

= 1,2,.··, M + N + 1) can

be updated as follows:

Wij (t + 1) =

wij (t)

+ fJ2Y i (t + l)vj (t),

(8.24)

where fh is the learning rate. Now, define the instantaneous sum of squared errors for the hidden layer units as 1 N &(t) = e; (t), 2 ｫｾｬ＠

L

(8.25)

where ek(t) is the difference (error) in the output of the kth hidden unit before and after the weight updated as 148

wij

is updated. Then, the instantaneous weight is

Chapter 8

Signal Reconstruction in Noisy Distorted Channels

_ wij(t+1) = wij(t+1)-J33

O&(t)

,

ｯｷｾＨｴＫＱＩ＠

(8.26)

where /33 is learning rate. From Eq. (8.16) and Eq. (8.25), we have

= _ ｾ＠

() oq; (t + 1)

L..ek t

k=1

where

ｯｷｾ＠

(t + 1)

(8.27)

,

q; (t + 1) is the output of the kth hidden unit after the weight

updated to ｷｾ＠

(t + 1). To determine the partial derivative oq; (t + 1) / ｯｷｾ＠

we differentiate Eq. (8.15) and Eq. (8.16) with respect to

wij

is

(t + 1),

By applying the

wij.

chain rule, we obtain oq; (t + 1) ork- (t + 1)

oq;(t + 1) ｯｷｾＨｴ＠

ork- (t + 1) ｯｷｾ＠

+ 1)

= m'(r - (t 'f'

(t + 1)

rk-.....:(t_+_1-,---) + 1)) _0--2

(8.28)

ｯｷｾＨｴＫＱＩＧ＠

k

where rk- (t + 1) corresponds to the updated internal state of hidden unit k. By using Eq.(8.16), we get ｯｷｾＨｴＫＱＩ＠

I

0

L..

_( 1) wkn t +

ork-(t+1) = M +1 n=1 ｯｷｾＨｴＫＱＩ＠ =

ｍｾＫｉ｛＠

n=1

[w- (t+1)v (t)J kn n

ｯｷｾＨｴ＠

ovn(t) ＨＩｏｗ［ｮｴＫＱｾ＠ +v t + 1) n ｯｷｾＨｴ＠

+ 1)

.

(8.29) · . ow;;;, (t + 1) IS . equato I one 1'f k = 1. an d n =j,. an d'IS Note that t h e d envatlve ｯｷｾＨｴＫＱＩ＠

zero otherwise. Thus, we may rewrite Eq. (8.29) as

ｯｷｾＨｴＫＱＩ＠

ｯｲｫＭＨｴＫＱＩ｟ｍｾｉ｛＠

-

L..

n=1

-(t + 1) ovn(t)

W kn

ｯｷｾＨｴＫＱＩ＠

1

s: (t) • + Uk V. I

(8.30)

}

149

8.2

Training Algorithm

From Eg. (8.14), we have

o

vn (I)

=

{

ow::(/+l) lj

oq;(t) ow:: (t + 1)

0

for n = M + 2, ... , M + N + 1.

(8.31 )

lj

otherwise.

We may then combine Eg. (8.28), Eg. (8.30) and Eg. (8.31) to yield

(8.32)

where J is the Kronecker delta function. We now defme a dynamical system described by a triply indexed set of variables {Q; (I + I)}, where

ｑｾＨｴＫｬＩ］＠ lj

oq;(t) , Owij(1 + 1)

(8.33)

for } = 1, 2,··· , M + N + 1, i = 1, 2,··· , N, and n = M + 2,··· , M + N + 1. For each time step 1 and all appropriate n, i and}, the dynamics of the system so defined is governed by

(8.34)

Finally, the weight between the input layer and the hidden layer is updated by N

wij (I + 1) = wij (I + 1) + /33

L e (I + ｉＩｑｾ＠ k

(t + 1).

(8.35)

ｫｾｬ＠

The above procedure is repeatedly applied to all input sample pairs during the training stage. 150

Chapter 8

8.3

Signal Reconstruction in Noisy Distorted Channels

Simulation Study

In this section we simulate a chaos-based communication system, which is subject to channel distortion and additive white Gaussian noise. Our purpose is to test the ability of the proposed equalizer in combating the channel effects and noise.

8.3.1

Chaotic Signal Transmission

Two chaotic systems will be used to evaluate the performance of the proposed equalizer in this chapter. The first system is based on the HEmon map: Xl (t

+ 1) = 1- ｡ｬｸｾ＠

(t) + x2(t),}

(8.36)

x2 (t + 1) = a 2 xI (t),

where al and az are the bifurcation parameters fixed at 1.4 and 0.3, respectively. In particular, we select Xz as the transmitted signal, which guarantees that Eq. (8.5) holds. The second system is based on the Chua's circuit [138], which is rewritten as follows:

ｾｬ＠ Ｚ｡ＳｾＲ＠

where

a3

and

a4

-K(XI )),}

x2

-

Xl

X2

X3

=

a 4 x2 ,

(8.37)

+ X3 '

are respectively fixed at 9 and -100/7, and

K (-)

is a

piecewise-linear function given by _{mIXI +(mo -ml ) K(XI ) - mOxl mixi - (mo - ml )

for

Xl:;:: 1

for for

1Xl 1< 1

(8.38)

Xl (-1

with mo = -117 and ml = -217. The attractor has the largest Lyapunov exponent and the Lyapunov dimension equal to 0.23 and 2.13, 151

8.3

Simulation Study

respectively [139]. In this case, we select

X3

as the transmitted signal. This

choice of the transmitted signal guarantees that the dynamics of the transmission system can be reconstructed, i.e., there exist functions/and 'P such that Eq. (8.5) holds.

8.3.2

Filtering Effect of Communication Channels

Three channel models will be used to test the performance of the proposed equalizer in this chapter. The first two channels are linear channels which can be described in the z domain by the following transformation functions: channel I :

HI (z) = 1+ 0.5z- 1

(8.39)

channel II:

H 2 (z)=0.3+0.5z- 1 +0.3z- 2

(8.40)

These two channel models are widely used to evaluate the performance of equalizers in communication systems [294]. Let us consider the frequency responses of the channels. The amplitude-frequency and phase-frequency responses of the channels are shown in Fig. 8.4. It is worth noting that from Fig. 8.4 (a), channel II has a deep spectrum null at a normalized angular frequency of2.56, which is difficult to equalize by the usual LTE [264]. The third channel to be studied is a nonlinear channel, which is shown in Fig. 8.5. This channel is also widely used for testing the performance of equalizers [295]. The model can be described by (8.41 ) where a1 and a2 are channel parameters which are fixed at 0.2 and -0.1, respectively, ands is the output of the linear part of the channel which is given by set) = 0.3482 x(t) + 0.8704 x(t -1) + 0.3482 x(t - 2).

(8.42)

Thus, the transformation function of the linear component can be expressed as H(z) = 0.3482 + 0.8704 Z-l + 0.3482 152

Z-2.

(8.43)

Chapter 8

Signal Reconstruction in Noisy Distorted Channels

10 0 -10 -20

EO

'S -30

"

@ (J

-40 -50 -60 -70 -3

-2

-1 0 Nonnalized angular frequency

2

3

(a)

3,----------------------------------, 2

-2 ＭＳ｟ｵｾＷＲＱＰ＠

Normalized angular frequency (b)

Figure 8.4 Frequency responses of Channel I (dotted line) and ChannelII (solid line) according to Eq. (S.39) and Eq. (S.40). (a) Magnitude responses; (b) Phase responses y

Figure 8.5 Nonlinear channel model studied in this book

153

8.3

Simulation Study

20 18 16 14 ｾ＠

Ｎｾ＠

Z

::s'"

12 10

8 6 4 2 0_

3 Nonnalized angular frequency (a)

16 14 12 0)

"c:l

10

Z

a

bJJ

::s'"

8

6 4 2

o Nonnalized angular frequency (b)

Figure 8.6 Illustration of channel effects. FFT magnitude spectrum of signal versus normalized angular frequency for (a) transmitted signal

Xl

from Eq. (8.36), and

(b) received signal y after passing through Channel I and contaminated by noise at SNR= 10 dB

154

ChapterS

Signal Reconstruction in Noisy Distorted Channels

004

IFf' '",

0.3

0.1

ｾ＠

I'

/

0.2

,

ＮＭｾ＠

Ｏ［ＭＺＮｾ＠

"i

0 ｾＮ＠

-0.1 -0.2 -0.3

.I /

/

-004

-0.3

-0.2

-0.1

0.1

0 x,(t)

0.2

\ 0.3

004

(a) 004 0.3 0.2 0.1 0 ｾ＠

+

"

,/ !

-0.1 -0.2 -0.3

/

!

-004 -0.5 -0.6

-004

-0.2

0.2

004

sit)

(b) 0.6

004 0.2

I

"

.

0

. , .,..-.....

"-,

:

-0.2

"',:

"

-004 '"

-0.6

-0.8

-0.6

-004

-0.2

0 y(t)

0.2

OA

0.6

(e)

Figure 8.7 Illustration of channel effects. Return maps reconstructed from (a) the transmitted Xl from Eq. (8,36), (b) the distorted s(t), and (c) the received signal yet) after passing through Channel I and contaminated by noise at SNR = lO dB

155

8.3

Simulation Study

As an example to illustrate the channel effects, we consider a communication event, in which the transmitted signal

X2

generated from Eq. (8.36)

passes through Channel I. When the SNR is 10 dB, the fast Fourier transform spectra of the transmitted signal and the received signal are shown in Figs. 8.6 (a) and (b), from which we clearly observe the wideband property of the transmitted signal and the distortion caused by the channel. Furthermore, the return maps reconstructed from the transmitted signal, the distorted signal due to Channel I, and the noisy received signal are shown in Fig. 8.7. In our previous study [101], it has been shown that without an equalizer, the simple inverse system approach will give unacceptable performance even when the channel, besides AWGN, is an ideal allpass filter, i.e., h = 1.

8.3.3

Results

The equalization for each channel model consists of two stages. The first is the training or adaptation stage in which the equalizer makes use of some partially known sample pairs to adapt to the communication environment. When the training or adaptation is completed, actual communication commences. Determining the size of training sample sets is important. If the sets are too small, the equalizer cannot experience all states of the system in the given communication environment, leading to poor reconstruction. However, if the sets are too large, the training duration will be excessively long. In our simulations, we consider different sizes of the training sample sets, and examine the results in terms of the mean-square-error (MSE) as a function of the number of iterations (in this book, one iteration means that the equalizer is trained once with all training samples), where the MSE is defined as