The Nonlinear Workbook: Chaos, Fractals, Cellular Automata, Genetic Algorithms, Gene Expression Programming, Support Vector Machine, Wavelets, Hidden ... Java And Symbolicc++ Programs [6 ed.] 9814583472, 9789814583473

Citation preview

T H E

9084hc_9789814583466_tp.indd 1

NONLINEAR WORKBOOK

20/10/14 5:03 pm

May 2, 2013

14:6

BC: 8831 - Probability and Statistical Theory

This page intentionally left blank

PST˙ws

T H E

NONLINEAR WORKBOOK

Chaos, Fractals, Cellular Automata, Genetic Algorithms, Gene Expression Programming, Support Vector Machine, Wavelets, Hidden Markov Models, Fuzzy Logic with C++, Java and SymbolicC++ Programs

6th Edition

Willi-Hans Steeb

International School for Scientific Computing University of Johannesburg, South Africa in collaboration with

Yorick Hardy

International School for Scientific Computing, South Africa

Ruedi Stoop

Institute for Neuroinformatics, University of Zürich/ETHZ

World Scientific NEW JERSEY

•

LONDON

9084hc_9789814583466_tp.indd 2

•

SINGAPORE

•

BEIJING

•

SHANGHAI

•

HONG KONG

•

TA I P E I

•

CHENNAI

20/10/14 5:03 pm

Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

Library of Congress Cataloging-in-Publication Data Steeb, W.-H. The nonlinear workbook : chaos, fractals, cellular automata, genetic algorithms, gene expression programming, support vector machine, wavelets, hidden Markov models, fuzzy logic with C++, java and symbolic C++ programs / Willi-Hans Steeb, University of Johannesburg, South Africa. -- 6th edition. pages cm Includes bibliographical references and index. ISBN 978-9814583466 (hardcover : alk. paper) -- ISBN 978-9814583473 (pbk. : alk. paper) 1. Nonlinear programming. 2. Nonlinear theories. I. Title. T57.8.S74 2015 519.7'6--dc23 2014040593

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

Copyright © 2015 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.

Printed in Singapore

Lakshmi - The nonlinear workbook (6th Ed).indd 1

17/10/2014 1:53:45 PM

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6

Preface The study of chaos, fractals, cellular automata, neural networks, genetic algorithms and fuzzy logic is one of the most fascinating subjects in science. Most of these fields are interrelated. Chaotic attractors are used in neural networks. Genetic algorithms can be used to train neural networks. Fractals are used in data compression. Neural networks and fuzzy logic are often combined when the input values of the system are not crisp. In this book we give all the basic concepts in these fields together with the definitions, theorems and algorithms. The algorithms are implemented using C++, Java and SymbolicC++. The level of presentation is such that one can study the subject early on in one’s education in science. There is a balance between practical computation and the underlying mathematical theory. In chapter 1, we consider one- and two-dimensional nonlinear maps. All the relevant quantities to characterize chaotic systems are introduced. Algorithms are given for all the quantities which are used to describe chaos such as invariant density, Liapunov exponent, correlation integral, autocorrelation function, capacity, phase portrait, Poincar´e section, Fourier transform, calculations of exact trajectories, fixed points and their stability, etc.. Sarkovskii’s theorem is introduced. Chaotic repellers and encoding using one-dimensional chaotic maps are also investigated. Newton’s method in one and two dimensions is derived. Periodic orbits and topological degree are introduced. The ergodic theorem is applied to Baker’s map. Quite often a dynamical system cannot be modelled by difference equations or differential equations, but an experiment provides a time series. In chapter 2, we consider quantities for the study of chaotic time-series. We also include the Hurst exponent which plays an important role in the study of financial markets. The related Higuchi algorithm is also provided. Brownian motion is also covered. In chapter 3, we first describe the classification of fixed points in the plane. Furthermore the most important two-dimensional dynamical systems are studied, such as the pendulum, limit cycle systems and a Lotka-Volterra model. Homoclinic orbits are also introduced. Several biochemical equations are introduced. Chapter 4 reviews integrable and chaotic Hamilton systems. Among other concepts v

page 1

October 17, 2014

vi

17:19

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6-ppvi-viii

PREFACE

we introduce the Lax representation for integrable Hamilton systems, the Poincar´e section and the Floquet theory. In chapter 5, nonlinear dissipative systems are studied. The most famous dissipative system with chaotic behaviour, the Lorenz model, is introduced. We also discuss Hopf bifurcation and hyperchaotic systems. Nonlinear driven systems play a central role in engineering, in particular, in electronics. In most cases the driving force is periodic. Chapter 6 is devoted to these systems. As examples we consider among others the driven pendulum and the driven van der Pol equation. The concept of torsion number is also discussed. Controlling of chaotic systems is very important in applications in engineering. In chapter 7, we discuss the different concepts of controlling chaos. The Ott-GrebogiYorke method for controlling chaotic systems is studied in detail. Synchronization of chaotic systems is described in chapter 8 and a number of applications are given, such as the coupled Rikitake dynamos. Fractals have become of increasing interest, not only in art, but also in many different areas of science such as compression algorithms. In chapter 9, we introduce iterated function systems, the Mandelbrot set, the Julia set and the Weierstrass function. The famous Cantor set is considered as an example as well as the Koch curve, fern and the Sierpinski gasket. We also derive the construction of fractals using the Kronecker product of matrices. Grey level maps are also described. Cellular automata are discrete dynamical systems. We describe in chapter 10 oneand two-dimensional cellular automata. The famous game of life with a C++ implementation and the button game with a Java implementation are also considered. The Sznajd model is studied as an application. Chapter 11 is about integration of ordinary differential equations. We describe the Euler method, the Runge-Kutta method, the Lie series technique, symplectic integration, Verlet method, etc.. Furthermore we discuss ghost solutions, invisible chaos and integration in the complex domain. Optimization problems are studied in chapter 12. We consider the Lagrange multiplier method for optimization problems and also describe an alternative method using differential forms. Cartan forms are introduced and applications to classical mechanics are described. For problems with inequality constraints the KarushKuhn-Tucker condition is provided and the support vector machine is studied.

page 1

October 17, 2014

17:19

PREFACE

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6-ppvi-viii

vii

The Kernel-Adatron algorithm, a fast and simple learning procedure for support vector machines, is also implemented. As an application the kernel Fisher discriminant is studied. A short introduction to particle swarm optimization is also given. Chapter 13 is devoted to neural networks. We introduce the Hopfield algorithm, the Kohonen self-organizing map, the back propagation algorithm and radial basis function networks. One of the applications is the traveling salesman problem. Neural oscillator models are also introduced. Genetic algorithms are used to solve optimization problems. Chapter 14 is devoted to this technique. We discuss optimization problems with and without constraints. Utilizing bitwise operations for genetic algorithm is discussed in detail. We also study simulated annealing. Gene expression programming is a new genetic algorithm that uses encoded individuals. Gene expression programming individuals are encoded in linear chromosomes which are expressed or translated into expression trees. The linear chromosome is the genetic material that is passed on with modifications to the next generation. Chapter 15 gives an introduction to this technique together with a C++ program. As an alternative to gene expression programming we also describe multi-expression programming together with a C++ program. Wavelet theory is a form of mathematical transformation, similar to the Fourier transform in that it takes a signal in time domain, and represents it in frequency domain. Wavelet functions are distinguished from other transformations in that they not only dissect signals into their component frequencies, they also vary the scale at which the component frequencies are analyzed. Chapter 16 provides an introduction. Filtering is given as an example application. We focus mainly on the Haar wavelet and Daubechies wavelet but other wavelets are also studied. Furthermore, two-dimensional wavelets are also considered. Discrete Hidden Markov Models are introduced in chapter 17. The forward-backward algorithm, Viterbi algorithm, and Baum-Welch algorithm are described. The application concentrates on speech recognition and DNA applications. Since its inception 40 years ago the theory of fuzzy sets has advanced in a variety of ways and in many disciplines, not only in science. Chapter 18 is devoted to fuzzy logic. Fuzzy numbers and arithmetic are also considered. Furthermore, decision making problems and controlling problems using fuzzy logic are also described. Fuzzy clustering is also included as well as a definition for the fuzzy XOR. In each chapter we give C++, Java and SymbolicC++ implementations of the algorithms. Furthermore, each chapter includes a number of stimulating and exciting exercises for the respective topic.

page 2

October 17, 2014

17:19

BC: 9084 – The Nonlinear Workbook (6th edition)

viii

9084-book9x6-ppvi-viii

PREFACE

Without doubt, this book can be extended. Any comments or suggestions are welcome. The author can be contacted via e-mail: [email protected] [email protected] The web page of the author is: http://issc.uj.ac.za The International School for Scientific Computing (ISSC) provides certificate courses for these subjects. Please contact the author if you want to do any of these courses.

page 3

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6

Contents Preface

v

Symbol Index

xvii

Comments to the Programs

xix

1 Nonlinear and Chaotic Maps 1.1 One-Dimensional Maps . . . . . . . . . . . . . . . . . . . 1.1.1 Exact and Numerical Trajectories . . . . . . . . . 1.1.2 Fixed Points and Stability . . . . . . . . . . . . . 1.1.3 Invariant Density . . . . . . . . . . . . . . . . . . 1.1.4 Liapunov Exponent . . . . . . . . . . . . . . . . . 1.1.5 Autocorrelation Function . . . . . . . . . . . . . . 1.1.6 Sarkovskii’s Theorem . . . . . . . . . . . . . . . . 1.1.7 Discrete One-Dimensional Fourier Transform . . . 1.1.8 Fast Fourier Transform . . . . . . . . . . . . . . . 1.1.9 Logistic Map and Liapunov Exponent for r ∈ [3, 4] 1.1.10 Logistic Map and Bifurcation Diagram . . . . . . . 1.1.11 Random Number Map and Invariant Density . . . 1.1.12 Random Number Map and Random Integration . . 1.1.13 Circle Map and Rotation Number . . . . . . . . . 1.1.14 One-Dimensional Newton Method . . . . . . . . . 1.1.15 Feigenbaum’s Constant . . . . . . . . . . . . . . . 1.1.16 Symbolic Dynamics . . . . . . . . . . . . . . . . . 1.1.17 Chaotic Repeller . . . . . . . . . . . . . . . . . . . 1.1.18 Chaos and Encoding . . . . . . . . . . . . . . . . . 1.1.19 Chaotic Data Communication . . . . . . . . . . . 1.1.20 Exercises . . . . . . . . . . . . . . . . . . . . . . . 1.2 Two-Dimensional Maps . . . . . . . . . . . . . . . . . . . 1.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . 1.2.2 Phase Portrait . . . . . . . . . . . . . . . . . . . . 1.2.3 Fixed Points and Stability . . . . . . . . . . . . . 1.2.4 Liapunov Exponents . . . . . . . . . . . . . . . . . 1.2.5 Correlation Integral . . . . . . . . . . . . . . . . . 1.2.6 Capacity . . . . . . . . . . . . . . . . . . . . . . . ix

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 4 15 16 21 24 26 27 29 34 35 37 39 41 43 46 47 50 51 56 59 60 60 64 70 72 73 75

page 5

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6

x

CONTENTS

1.3

1.2.7 Hyperchaos . . . . . . . . . . . . . . . . . 1.2.8 Domain of Attraction . . . . . . . . . . . 1.2.9 Newton Method in the Complex Domain 1.2.10 Newton Method in Higher Dimensions . . 1.2.11 Ruelle-Takens-Newhouse Scenario . . . . 1.2.12 Melnikov Analysis for Maps . . . . . . . . 1.2.13 Periodic Orbits and Topological Degree . 1.2.14 JPEG file . . . . . . . . . . . . . . . . . . 1.2.15 Baker’s Map and Ergodic Theorems . . . 1.2.16 Fixed Point Theorems . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

77 80 82 84 86 88 89 91 93 95 96

2 Time Series Analysis 2.1 Introduction . . . . . . . . . . . . . . . . . . . 2.2 Correlation Coefficient . . . . . . . . . . . . . . 2.3 Liapunov Exponent from Time Series . . . . . 2.3.1 Jacobian Matrix Estimation Algorithm 2.3.2 Direct Method . . . . . . . . . . . . . . 2.4 Hurst Exponent . . . . . . . . . . . . . . . . . 2.4.1 Introduction . . . . . . . . . . . . . . . 2.4.2 Implementation of the Hurst Exponent 2.5 Random Walk . . . . . . . . . . . . . . . . . . 2.6 Brownian Motion . . . . . . . . . . . . . . . . 2.7 Higuchi’s Algorithm . . . . . . . . . . . . . . . 2.8 Complexity . . . . . . . . . . . . . . . . . . . . 2.9 Exercises . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

97 97 98 100 100 102 108 108 110 114 119 121 122 126

3 Autonomous Systems in the Plane 3.1 Classification of Fixed Points . . 3.2 Homoclinic Orbit . . . . . . . . . 3.3 One-Dimensional Pendulum . . . 3.4 Limit Cycle Systems . . . . . . . 3.5 Lotka-Volterra Systems . . . . . 3.6 Biochemical Equations . . . . . . 3.7 Exercises . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

127 127 130 132 134 138 141 142

4 Nonlinear Hamilton Systems 4.1 Hamilton Equations of Motion . . . . . . . . . . . . . . . 4.1.1 Hamilton System and Variational Equation . . . . 4.2 Integrable Hamilton Systems . . . . . . . . . . . . . . . . 4.2.1 Hamilton Systems and First Integrals . . . . . . . 4.2.2 Lax Pair and Hamilton Systems . . . . . . . . . . 4.2.3 Floquet Theory . . . . . . . . . . . . . . . . . . . 4.3 Chaotic Hamilton Systems . . . . . . . . . . . . . . . . . 4.3.1 Trajectories and H´enon-Heiles Hamilton Function

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

143 143 146 147 147 150 152 155 155

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

page 6

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6

CONTENTS

4.4

xi

4.3.2 Surface of Section Method . . . . . . . . . . . . . . . . . . . 157 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

5 Nonlinear Dissipative Systems 5.1 Fixed Points and Stability . . . . . 5.2 Trajectories . . . . . . . . . . . . . 5.3 Phase Portrait . . . . . . . . . . . 5.4 Liapunov Exponents . . . . . . . . 5.5 Generalized Lotka-Volterra Model 5.6 Hyperchaotic Systems . . . . . . . 5.7 Hopf Bifurcation . . . . . . . . . . 5.8 Time-Dependent First Integrals . . 5.9 Exercises . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

163 163 168 172 174 177 179 182 184 186

6 Nonlinear Driven Systems 6.1 Introduction . . . . . . . . . . . . . . . . . . . . 6.2 Driven Anharmonic Systems . . . . . . . . . . . 6.2.1 Phase Portrait . . . . . . . . . . . . . . . 6.2.2 Poincar´e Section . . . . . . . . . . . . . . 6.2.3 Liapunov Exponent . . . . . . . . . . . . 6.2.4 Autocorrelation Function . . . . . . . . . 6.2.5 Power Spectral Density . . . . . . . . . . 6.3 Driven Pendulum . . . . . . . . . . . . . . . . . 6.3.1 Phase Portrait . . . . . . . . . . . . . . . 6.3.2 Poincar´e Section . . . . . . . . . . . . . . 6.4 Parametrically Driven Pendulum . . . . . . . . . 6.4.1 Phase Portrait . . . . . . . . . . . . . . . 6.4.2 Poincar´e Section . . . . . . . . . . . . . . 6.5 Driven Van der Pol Equation . . . . . . . . . . . 6.5.1 Phase Portrait . . . . . . . . . . . . . . . 6.5.2 Liapunov Exponent . . . . . . . . . . . . 6.6 Parametrically and Externally Driven Pendulum 6.7 Torsion Numbers . . . . . . . . . . . . . . . . . . 6.8 Exercises . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

187 187 190 190 191 193 194 197 198 198 199 201 201 203 204 204 206 207 210 214

7 Controlling of Chaos 7.1 Introduction . . . . . . . . . . . . . . . 7.2 Ott-Yorke-Grebogi Method . . . . . . . 7.2.1 One-Dimensional Maps . . . . . 7.2.2 Systems of Difference Equations 7.3 Time-Delayed Feedback Control . . . . 7.4 Small Periodic Perturbation . . . . . . . 7.5 Resonant Perturbation and Control . . 7.6 Exercises . . . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

215 215 215 215 219 222 225 227 228

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . . . .

. . . . . . . .

page 7

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6

xii

CONTENTS

8 Synchronization of Chaos 8.1 Introduction . . . . . . . . . . . . . . 8.2 Synchronization of Chaos . . . . . . . 8.2.1 Synchronization Using Control 8.2.2 Synchronizing Subsystems . . . 8.3 Synchronization of Coupled Dynamos 8.4 Phase Coupled Systems . . . . . . . . 8.5 Exercises . . . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

229 229 229 229 232 235 236 242

9 Fractals 9.1 Introduction . . . . . . . . . . . . 9.2 Fractal Dimensions . . . . . . . . . 9.3 Iterated Function System . . . . . 9.3.1 Introduction . . . . . . . . 9.3.2 Cantor Set . . . . . . . . . 9.3.3 Heighway’s Dragon . . . . 9.3.4 Sierpinski Gasket . . . . . 9.3.5 Koch Curve . . . . . . . . 9.3.6 Fern . . . . . . . . . . . . . 9.3.7 Grey Level Maps . . . . . . 9.4 Mandelbrot Set . . . . . . . . . . . 9.5 Julia Set . . . . . . . . . . . . . . 9.6 Fractals and Kronecker Product . 9.7 Lindenmayer Systems and Fractals 9.8 Weierstrass Function . . . . . . . . 9.9 L´evy-Flight Random Walk . . . . 9.10 Exercises . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

243 243 244 245 245 246 249 251 253 255 256 257 259 261 265 268 270 271

10 Cellular Automata 10.1 Introduction . . . . . . . . . . . . . . 10.2 A Spin System and Cellular Automata 10.3 Sznajd Model . . . . . . . . . . . . . . 10.4 Conservation Laws . . . . . . . . . . . 10.5 Two-Dimensional Cellular Automata . 10.6 Button Game . . . . . . . . . . . . . . 10.7 Langton’s Ant . . . . . . . . . . . . . 10.8 Quantum Cellular Automata . . . . . 10.9 Exercises . . . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

273 273 276 277 280 281 285 288 290 292

11 Solving Ordinary Differential Equations 11.1 Introduction . . . . . . . . . . . . . . . 11.2 Euler Method . . . . . . . . . . . . . . 11.3 Lie Series Technique . . . . . . . . . . . 11.4 Runge-Kutta-Fehlberg Technique . . . . 11.5 Ghost Solutions . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

293 293 295 297 301 302

. . . . . . . . . . . . . . . . .

page 8

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6

CONTENTS 11.6 11.7 11.8 11.9 11.10 11.11

Symplectic Integration . . . . Verlet Method . . . . . . . . St¨ormer Method . . . . . . . Invisible Chaos . . . . . . . . First Integrals and Numerical Exercises . . . . . . . . . . .

xiii . . . . . . . . . . . . . . . . . . . . . . . . . . . . Integration . . . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

305 310 311 312 313 315

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

317 317 319 323 325 330 335 335 335 338 342 346 349

13 Neural Networks 13.1 Introduction . . . . . . . . . . . . . . . . . . . . 13.2 Hopfield Model . . . . . . . . . . . . . . . . . . . 13.2.1 Introduction . . . . . . . . . . . . . . . . 13.2.2 Synchronous Operations . . . . . . . . . . 13.2.3 Energy Function . . . . . . . . . . . . . . 13.2.4 Basins and Radii of Attraction . . . . . . 13.2.5 Spurious Attractors . . . . . . . . . . . . 13.2.6 Hebb’s Law . . . . . . . . . . . . . . . . . 13.2.7 Hopfield Example . . . . . . . . . . . . . 13.2.8 Hopfield C++ Program . . . . . . . . . . 13.2.9 Asynchronous Operation . . . . . . . . . 13.2.10 Translation Invariant Pattern Recognition 13.3 Similarity Metrics . . . . . . . . . . . . . . . . . 13.4 Kohonen Network . . . . . . . . . . . . . . . . . 13.4.1 Introduction . . . . . . . . . . . . . . . . 13.4.2 Kohonen Algorithm . . . . . . . . . . . . 13.4.3 Kohonen Example . . . . . . . . . . . . . 13.4.4 Traveling Salesman Problem . . . . . . . 13.5 Perceptron . . . . . . . . . . . . . . . . . . . . . 13.5.1 Introduction . . . . . . . . . . . . . . . . 13.5.2 Boolean Functions . . . . . . . . . . . . . 13.5.3 Perceptron Learning . . . . . . . . . . . . 13.5.4 One and Two Layered Networks . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

351 351 355 355 357 359 360 361 361 363 364 368 369 370 375 375 375 378 383 386 386 388 390 396

12 Optimization 12.1 Lagrange Multiplier Method . . . . . 12.2 Examples . . . . . . . . . . . . . . . . 12.3 Coordinate Systems . . . . . . . . . . 12.4 Differential Forms and Cartan Forms . 12.5 Karush-Kuhn-Tucker Conditions . . . 12.6 Support Vector Machine . . . . . . . . 12.6.1 Introduction . . . . . . . . . . 12.6.2 Linear Decision Boundaries . . 12.6.3 Nonlinear Decision Boundaries 12.6.4 Kernel Fisher Discriminant . . 12.7 Particle Swarm Optimization . . . . . 12.8 Exercises . . . . . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

page 9

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6

xiv

CONTENTS 13.5.5 XOR Problem and Two Layered Networks . 13.6 Multilayer Perceptrons . . . . . . . . . . . . . . . . 13.6.1 Introduction . . . . . . . . . . . . . . . . . 13.6.2 Cybenko’s Theorem . . . . . . . . . . . . . 13.6.3 Back-Propagation Algorithm . . . . . . . . 13.7 Radial Basis Function Networks . . . . . . . . . . 13.8 Recursive Perceptron Neural Networks . . . . . . . 13.9 Chaotic Neural Networks . . . . . . . . . . . . . . 13.10 Neuronal-Oscillator Models . . . . . . . . . . . . . 13.11 Neural Network, Matrices and Eigenvalues . . . . . 13.12 Exercises . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

397 401 401 402 402 410 414 415 416 418 421

14 Genetic Algorithms 14.1 Introduction . . . . . . . . . . . . . . . . . 14.2 Sequential Genetic Algorithm . . . . . . . . 14.3 Schemata Theorem . . . . . . . . . . . . . 14.4 Bitwise Operations . . . . . . . . . . . . . . 14.4.1 Introduction . . . . . . . . . . . . . 14.4.2 Assembly Language . . . . . . . . . 14.4.3 Floating Point Numbers and Bitwise 14.4.4 C++ bitset Class . . . . . . . . . . 14.4.5 Java Bitset Class . . . . . . . . . . . 14.5 Penna Bit-String Model . . . . . . . . . . . 14.6 Maximum of One-Dimensional Maps . . . . 14.7 Maximum of Two-Dimensional Maps . . . . 14.8 Finding a Fitness Function . . . . . . . . . 14.9 Problems with Constraints . . . . . . . . . 14.9.1 Introduction . . . . . . . . . . . . . 14.9.2 Knapsack Problem . . . . . . . . . . 14.9.3 Traveling Salesman Problem . . . . 14.10 Simulated Annealing . . . . . . . . . . . . . 14.11 Exercises . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

423 423 425 428 430 430 432 435 436 438 439 441 446 453 460 460 461 464 474 476

15 Gene Expression Programming 15.1 Introduction . . . . . . . . . . . . 15.2 Examples . . . . . . . . . . . . . . 15.2.1 Maps . . . . . . . . . . . . 15.2.2 Boolean Expression . . . . 15.3 Numerical-Symbolic Manipulation 15.4 Multi Expression Programming . . 15.5 Exercises . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

479 479 482 482 488 493 498 503

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . . . . . .

. . . . . . .

. . . . . . . . . . .

. . . . . . .

. . . . . . .

page 10

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6

CONTENTS 16 Wavelets 16.1 Introduction . . . . . . . . . . . . . . . . 16.2 Haar Wavelet . . . . . . . . . . . . . . . . 16.3 Other Wavelets . . . . . . . . . . . . . . . 16.4 Multiresolution Analysis . . . . . . . . . . 16.5 Biorthogonal Wavelets . . . . . . . . . . . 16.6 Two-Dimensional Wavelets . . . . . . . . 16.7 Pyramid Algorithm and Discrete Wavelets 16.8 Exercises . . . . . . . . . . . . . . . . . .

xv

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

505 505 507 509 510 513 518 520 524

17 Discrete Hidden Markov Models 17.1 Introduction . . . . . . . . . . . . . . . . . . . . . 17.2 Stochastic Matrices . . . . . . . . . . . . . . . . . 17.3 Markov Chains . . . . . . . . . . . . . . . . . . . . 17.4 Discrete Hidden Markov Models . . . . . . . . . . 17.4.1 Definitions . . . . . . . . . . . . . . . . . . 17.4.2 The Three Problems . . . . . . . . . . . . . 17.4.3 Forward-Backward Algorithm . . . . . . . . 17.4.4 Viterbi Algorithm . . . . . . . . . . . . . . 17.4.5 Baum-Welch Algorithm . . . . . . . . . . . 17.4.6 Distances between Hidden Markov Models . 17.5 Applications of Hidden Markov Models . . . . . . 17.5.1 Speech Recognition . . . . . . . . . . . . . 17.5.2 Biological Systems . . . . . . . . . . . . . . 17.6 C++ Program . . . . . . . . . . . . . . . . . . . . 17.7 Exercises . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

525 525 528 532 534 534 536 538 541 542 542 544 544 547 552 560

18 Fuzzy Sets and Fuzzy Logic 18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . 18.2 Definitions and Examples . . . . . . . . . . . . . . . 18.3 Operators for Fuzzy Sets . . . . . . . . . . . . . . . 18.3.1 Logical Operators . . . . . . . . . . . . . . . 18.3.2 Algebraic Operators . . . . . . . . . . . . . . 18.3.3 Defuzzification Operators . . . . . . . . . . . 18.3.4 Fuzzy Concepts as Fuzzy Sets . . . . . . . . 18.3.5 Hedging . . . . . . . . . . . . . . . . . . . . . 18.3.6 Quantifying Fuzzyness . . . . . . . . . . . . . 18.3.7 C++ Implementation of Discrete Fuzzy Sets 18.4 Fuzzy Numbers and Fuzzy Arithmetic . . . . . . . . 18.4.1 Introduction . . . . . . . . . . . . . . . . . . 18.4.2 Algebraic Operations . . . . . . . . . . . . . 18.4.3 LR-Representations . . . . . . . . . . . . . . 18.4.4 Algebraic Operations on Fuzzy Numbers . . . 18.4.5 C++ Implementation of Fuzzy Numbers . . . 18.4.6 Applications . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

561 561 566 570 570 572 574 575 576 578 578 601 601 602 604 607 608 614

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

page 11

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

9084-book9x6

xvi

CONTENTS 18.5 Fuzzy Rule-Based Systems . . . . . . . . . . . 18.5.1 Introduction . . . . . . . . . . . . . . . 18.5.2 Fuzzy If-Then Rules . . . . . . . . . . . 18.5.3 Inverted Pendulum Control System . . 18.5.4 Fuzzy Controllers with B-Spline Models 18.5.5 Application . . . . . . . . . . . . . . . . 18.6 Fuzzy C-Means Clustering . . . . . . . . . . . 18.7 T-Norms and T-Conorms . . . . . . . . . . . . 18.8 Fuzzy Logic Networks . . . . . . . . . . . . . . 18.9 Fuzzy Hamming Distance . . . . . . . . . . . . 18.10 Fuzzy Truth Values and Probabilities . . . . . 18.11 Exercises . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

615 615 619 620 621 624 626 630 631 633 637 638

Bibliography

639

Index

657

page 12

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

Symbol Index ∅ A⊂B A∩B A∪B χA N N0 Z Q R R+ C Rn Cn i 0 the forward trajectory tends to 1. The fixed points of f are 0 and 1. ♣ Example. Consider the map f : [0, 1] → [0, 1], f (x) = x2 . If x0 = 0, then f (x0 ) = 0. Analogously, if x0 = 1, then f (x0 ) = 1. The points 0 and 1 are fixed points for this map. For x ∈ (0, 1), the forward trajectory tends to 0. ♣ In most cases the behaviour of a map is much more complex. Example. Consider the map f : [0, 1] → [0, 1] with f (x) = 4x(1 − x). Then x∗ = 3/4 and x∗ = 0 are fixed points. The preimage of the fixed point x∗ = 3/4 is obtained by solving the quadratic equation 4x(1 − x) = 3/4. It provides the two solution x1 = 3/4 and x2 = 1/4. x1 is obvious since 3/4 is a fixed point. ♣ Definition. A point x∗ ∈ S is called a periodic point of period n if f (n) (x∗ ) = x∗ where f (n) denotes the n-th iterate of f . The least positive integer n for which f (n) (x∗ ) = x∗ is called the prime period of x∗ . The set of all iterates of a periodic point form a periodic orbit. Example. Consider the map f : R → R and f (x) = x2 − 1. Then the points 0 and −1 lie on a periodic orbit of period 2 since f (0) = −1 and f (−1) = 0. ♣ Definition. A point x∗ is eventually periodic of period n if x∗ is not periodic but there exists m > 0 such that f (n+j) (x∗ ) = f (j) (x∗ ) for all j ≥ m. That is, f (j) (x∗ ) is periodic for j ≥ m.

√ Example. Consider the map f : R → R with f (x) = x2 −1. Then with x0 = 2 we have the orbit x1 = 1, x2 = 0, x3 = −1, x4 = 0, i.e. the orbit is eventually periodic.♣ Definition. Let x∗ be a periodic point of prime period n. The point x∗ is hyperbolic if |(f (n) )0 (x∗ )| 6= 1 where 0 denotes the derivative of the n-th iterate f (n) with respect to x.

page 18

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

1.1. ONE-DIMENSIONAL MAPS

9084-book9x6

3

2 Example. Consider the map fr : R p→ R with fr (x) = x 0 + r and r ∈ R. Then ∗ the fixed points are √ x± = 1/2 ± 1/4 − r. We have fr ≡ dfr /dx = 2x and dfr (x∗+ )/dx = 1 + 1 − 4r. With r = 1/4 we have |fr0 (x∗+ )| = 1 and the fixed point is non-hyperbolic. ♣

Theorem. Let x∗ be a hyperbolic fixed point of a differentiable one-dimensional map f with |f 0 (x∗ )| < 1. Then there is an open interval U about x∗ such that if x ∈ U , then lim f (n) (x) = x∗ . n→∞

Example. Consider the map f : R → R and f (x) = 2x(1 − x). Then the fixed point x∗ = 1/2 is stable since |f 0 (1/2)| = 0. The point 1/3 is in the open interval U about x∗ = 1/2 such that f (n) (1/3) tends to 1/2. ♣ Definition. Let M be a differentiable manifold. A mapping f : M → M is called a diffeomorphism if f is a bijection with f and f −1 continuously differentiable (of class C 1 ). Example. The map f : R → R, f (x) = sinh(x) is a diffeomorphism. The only fixed point of the map is x∗ = 0. Since df (x)/dx = cosh(x) and cosh(x) ≥ 1 for all x ∈ R the map has no critical points. ♣ Example. The map f : R → R, f (x) = x3 is not a diffeomorphism since its derivative vanishes at 0. The fixed points of the map are −1, 0, 1. Since df (x∗ = 0)/dx = 0 we find that the fixed point x∗ = 0 is stable. ♣ We introduce the following concepts important in the study of nonlinear and chaotic maps 1) Fixed points and their stability 2) Liapunov exponent 3) Invariant density 4) Autocorrelation functions 5) Moments 6) Sarkovskii’s theorem 7) Discrete Fourier transform 8) Bifurcation diagrams 9) Feigenbaum number 10) Symbolic dynamics 11) Chaotic repeller The one-dimensional maps we study in the examples are the logistic map, the tent map, the Bernoulli map, the Gauss map, a bungalow-tent map and the circle map. A necessary condition for a one-dimensional map to show chaotic behaviour is that the map is non-invertible.

page 19

October 16, 2014

17:7

BC: 9084 – The Nonlinear Workbook (6th edition)

4

9084-book9x6

CHAPTER 1. NONLINEAR AND CHAOTIC MAPS

1.1.1

Exact and Numerical Trajectories

In this section we calculate trajectories for one-dimensional maps. In the first example we consider the map f : N → N defined by  n/2 if n is even f (n) := 3n + 1 if n is odd where N denotes the natural numbers. For this map it is conjectured that for all initial values the trajectory finally tends to the periodic orbit ... 4 2 1 4 2 1 .... The data type unsigned long (4 bytes) in C++ is restricted to the range 0...4294967295 and the data type long (4 bytes) in C++ is restricted to the range -2147483648...+2147483647 To check the conjecture for larger initial values we use the abstract data type Verylong in SymbolicC++. In this class all arithmetic operators are overloaded. The overloaded operators are +,

-,

*,

/,

%,

+=,

-=,

*=,

/=

.

For example for the initial value 28 we find the sequence 14, 7, 22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5, 16, 8, 4, 2, 1, ... Thus the orbit is eventually periodic. Two different initial values are considered in the program trajectory1.cpp, namely 28 and 998123456789. // trajectory1.cpp #include #include "verylong.h" using namespace std;

// for cout // for data type Verylong of SymbolicC++

int main(void) { unsigned long n = 28; // initial value unsigned long T = 20; // number of iterations unsigned long t; for(t=0;t